Render video to MTLTexture

Question

Created Nov ’16

Replies 3

Boosts 0

Views 2.3k

Participants 2

I'm trying to render video to a Metal texture.

I followed this as the base of my class: https://github.com/FlexMonkey/VideoEffects/blob/master/VideoEffects/FilteredVideoVendor.swift

It basically opens a video using AVPlayer, and it creates a CIContext to work with the frames.

And on initialization, I create a texture like this,

let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm, width: 256, height: 256, mipmapped: false)
texture = device.makeTexture(descriptor: desc)

In every step of the CADisplayLink function I call this,

    func renderMetalTexture() {
        guard let img = unfilteredImage,
            let tex = texture else {
            return
        }
        let rect = CGRect(x: 0, y: 0, width: 256, height: 256)
        ciContext.render(img, to: tex, commandBuffer: nil, bounds: rect, colorSpace: CGColorSpace.sRGB as! CGColorSpace)
    }

When I try to execute this code, the application crashes without much explanation.

EXC_BREAKPOINT (code=1, subcode=0x1000fb578)

Any idea of what's going on? I couldn't find any help code. The only thing I could find are the slides of session 510 from WDC2015 but the only explanation there is that the commandBuffer can be nil...

Boost

Answer 1

wcm OP

Nov ’16

How are you creating your

CIContext

? If you're rendering to a Metal texture with

render(_:to:commandBuffer:bounds:colorSpace:)

, you're obligated to initialize your

CIContext

with an

MTLDevice

, regardless of whether you provide your own command buffer.

0

Answer 2

endavid OP

Nov ’16

I wasn't passing the MTLDevice. I fixed that, but it still crashes at the same place. The subcode is different now, though,

EXC_BREAKPOINT (code=1, subcode=0x100087948)

0

Answer 3

endavid OP

Nov ’16

The next reason for the crash was the color space.

I've added this to fix the crash,

        let colorSpace = img.colorSpace ?? CGColorSpaceCreateDeviceRGB()

And I pass that to render. Now it doesn't crash and it renders 🙂

However, the size of the video is wrong... It's supposed to be 1080x1080, but if I inspect the img.extent, it says it's 1920x1080. I took a GPU frame capture and indeed it looks like the aspect has changed 😟

0