CVMetalTextureCacheCreateTextureFromImage use count note and how to handle it

The documentation CVMetalTextureCacheCreateTextureFromImage has the following "Important" note:

This function increments the use count of the image buffer, but not the IOSurface buffer. The Core Video Metal texture owns this IOSurface buffer. Therefore, you must maintain a strong reference to either the image buffer or the Metal texture until the Metal rendering completes. Consider using the MTLCommandBufferHandler for this purpose.

However, when using CFGetRetainCount on the CVPixelBufferRef the count remains the same.

Furthermore, the MTLTexture aquired from the returned CVMetalTextureRef has a strong reference to the IOSurfaceRef, so unless when using commandBufferWithUnretainedReferences there should be no risk that the backing storage will be discarded before the command buffer completes.

So what does this note mean? Am I responsible for decrementing the use count of the passed CVPixelBufferRef? Do I even need to do something with the underlying IOSurfaceRef?

Post not yet marked as solved Up vote post of ishaked Down vote post of ishaked
328 views
Posted by
ishaked
Reply
Add a Comment

Replies

CV* API's are part of CoreVideo, so you'll likely want to tag this question with CoreVideo to get the full scoop.

Posted by
Media Engineer
Up vote reply of Media Engineer Down vote reply of Media Engineer

  • Can I add a tag after the fact? I don't see such an option, not an option to edit the question :/

    —
    ishaked
Add a Comment

The use count refers to the IOSurface's usage by an API. This differs from its retain count. If the use count is non-zero the IOSurface will not be recycled by a CVPixelBufferPool that created it. This prevents race conditions where one API writes to the surface while another simultaneously reads from it.  If there is a CVMetalTextureRef, CVImageBufferRef, CVPixelBufferRef the underlying IOSurface's use count will be non-zero. 

The note is trying to say that the surface should stay inUse until the command buffer is done with it.   To do this, you need to retain the CVMetalTextureRef until the kernel schedules any command buffer using and fires its schedule handler.

Posted by
Graphics and Games Engineer
Post not yet marked as solved Up vote reply of Graphics and Games Engineer Down vote reply of Graphics and Games Engineer

  • Thank you for this clarification. Regarding the final note "you need to retain the CVMetalTextureRef..." does retaining the MTLTexture obtained by CVMetalTextureCacheCreateTextureFromImage suffice? Otherwise - is the code in this example, in the method - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection wrong? It seems to retain the resulting MTLTexture, but the CVMetalTextureRef is release, before any GPU work is scheduled on the texture.

    https://developer.apple.com/library/archive/samplecode/MetalVideoCapture/Listings/MetalVideoCapture_AAPLRenderer_mm.html#//apple_ref/doc/uid/TP40015131-MetalVideoCapture_AAPLRenderer_mm-DontLinkElementID_8

    —
    ishaked

  • You need to keep a retain the CVMetalTextureRef until command buffers using it have been scheduled. Retaining an id<MTLTexture> is not sufficient to prevent it from being recycled.

    We looking into what the sample is doing a little bit and it looks like it might not be doing the right thing. (Unfortunately, the sample is pretty old and has been archived so we aren't likely to publish an update).

    —
    Graphics and Games Engineer
Add a Comment