I am drawing stuff onto an off-screen MTLTexture. (using Skia Canvas)
At a later point, I want to render this MTLTexture into a CAMetalLayer to display it on the screen.
Since I was using Skia for the off-screen drawing operations, my code is quite simple and I don't have the typical Metal setup (no MTLLibrary, MTLRenderPipelineDescriptor, MTLRenderPassDescriptor, MTLRenderEncoder, etc).
I now simply want to draw that MTLTexture into a CAMetalLayer, but haven't figured out how to do so simply.
This is where I draw my stuff to the MTLTexture _texture (Skia code):
- (void) renderNewFrameToCanvas(Frame frame) {
if (_skContext == nullptr) {
GrContextOptions grContextOptions;
_skContext = GrDirectContext::MakeMetal((__bridge void*)_device,
// TODO: Use separate command queue for this context?
(__bridge void*)_commandQueue,
grContextOptions);
}
@autoreleasepool {
// Lock Mutex to block the runLoop from overwriting the _texture
std::lock_guard lockGuard(_textureMutex);
auto texture = _texture;
// Get & Lock the writeable Texture from the Metal Drawable
GrMtlTextureInfo fbInfo;
fbInfo.fTexture.retain((__bridge void*)texture);
GrBackendRenderTarget backendRT(texture.width,
texture.height,
1,
fbInfo);
// Create a Skia Surface from the writable Texture
auto skSurface = SkSurface::MakeFromBackendRenderTarget(_skContext.get(),
backendRT,
kTopLeft_GrSurfaceOrigin,
kBGRA_8888_SkColorType,
nullptr,
nullptr);
auto canvas = skSurface->getCanvas();
auto surface = canvas->getSurface();
// Clear anything that's currently on the Texture
canvas->clear(SkColors::kBlack);
// Converts the Frame to an SkImage - RGB.
auto image = SkImageHelpers::convertFrameToSkImage(_skContext.get(), frame);
canvas->drawImage(image, 0, 0);
// Flush all appended operations on the canvas and commit it to the SkSurface
canvas->flush();
// TODO: Do I need to commit?
/*
id<MTLCommandBuffer> commandBuffer([_commandQueue commandBuffer]);
[commandBuffer commit];
*/
}
}
Now, since I have the MTLTexture _texture in memory, I want to draw it to the CAMetalLayer _layer. This is what I have so far:
- (void) setup {
// I set up a runLoop that calls render() 60 times a second.
// [removed to simplify]
_renderPassDescriptor = [[MTLRenderPassDescriptor alloc] init];
// Load the compiled Metal shader (PassThrough.metal)
auto baseBundle = [NSBundle mainBundle];
auto resourceBundleUrl = [baseBundle URLForResource:@"VisionCamera" withExtension:@"bundle"];
auto resourceBundle = [[NSBundle alloc] initWithURL:resourceBundleUrl];
auto shaderLibraryUrl = [resourceBundle URLForResource:@"PassThrough" withExtension:@"metallib"];
id<MTLLibrary> defaultLibrary = [_device newLibraryWithURL:shaderLibraryUrl error:nil];
id<MTLFunction> vertexFunction = [defaultLibrary newFunctionWithName:@"vertexPassThrough"];
id<MTLFunction> fragmentFunction = [defaultLibrary newFunctionWithName:@"fragmentPassThrough"];
// Create a Pipeline Descriptor that connects the CPU draw operations to the GPU Metal context
auto pipelineDescriptor = [[MTLRenderPipelineDescriptor alloc] init];
pipelineDescriptor.label = @"VisionCamera: Frame Texture -> Layer Pipeline";
pipelineDescriptor.vertexFunction = vertexFunction;
pipelineDescriptor.fragmentFunction = fragmentFunction;
pipelineDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
_pipelineState = [_device newRenderPipelineStateWithDescriptor:pipelineDescriptor error:nil];
}
- (void) render() {
@autoreleasepool {
// Blocks until the next Frame is ready (16ms at 60 FPS)
auto drawable = [_layer nextDrawable];
std::unique_lock lock(_textureMutex);
auto texture = _texture;
MTLRenderPassDescriptor* renderPassDescriptor = [[MTLRenderPassDescriptor alloc] init];
renderPassDescriptor.colorAttachments[0].texture = drawable.texture;
renderPassDescriptor.colorAttachments[0].loadAction = MTLLoadActionClear;
renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor();
id<MTLCommandBuffer> commandBuffer([_commandQueue commandBuffer]);
auto renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
[renderEncoder setLabel:@"VisionCamera: PreviewView Texture -> Layer"];
[renderEncoder setRenderPipelineState:_pipelineState];
[renderEncoder setFragmentTexture:texture atIndex:0];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:drawable];
[commandBuffer commit];
lock.unlock();
}
}
And along with that, I have created the PassThrough.metal shader which is just for passing through a texture:
#include <metal_stdlib>
using namespace metal;
// Vertex input/output structure for passing results from vertex shader to fragment shader
struct VertexIO
{
float4 position [[position]];
float2 textureCoord [[user(texturecoord)]];
};
// Vertex shader for a textured quad
vertex VertexIO vertexPassThrough(const device packed_float4 *pPosition [[ buffer(0) ]], const device packed_float2 *pTexCoords [[ buffer(1) ]], uint vid [[ vertex_id ]]) {
VertexIO outVertex;
outVertex.position = pPosition[vid];
outVertex.textureCoord = pTexCoords[vid];
return outVertex;
}
// Fragment shader for a textured quad
fragment half4 fragmentPassThrough(VertexIO inputFragment [[ stage_in ]], texture2d<half> inputTexture [[ texture(0) ]], sampler samplr [[ sampler(0) ]]) {
return inputTexture.sample(samplr, inputFragment.textureCoord);
}
Running this crashes the app with the following exception:
validateRenderPassDescriptor:782: failed assertion `RenderPass Descriptor Validation
Texture at colorAttachment[0] has usage (0x01) which doesn't specify MTLTextureUsageRenderTarget (0x04)
This now raises three questions for me:
Do I have to do all of that Metal setting up, packing along the PassThrough.metal shader, render pass stuff, etc just to draw the MTLTexture to the CAMetalLayer? Is there no simpler way?
Why is the code above failing?
When is the drawing from Skia actually committed to the MTLTexture? Do I need to commit the command buffer (as seen in my TODO)?