Crash when enabling jittering on SCNRenderer

Question

I'm trying to export an SCNScene (with animations) to a video file. It's working, but the result is jagged, so I want to add jittering to the scene I'm rendering.

I'm adding one line of code that makes this Apple sample code crash.

Heres a direct link to the line I added - line 47

The line in question is

Code Block renderer.isJitteringEnabled = true

When I remove the line, it works fine.

When I add the line, the app crashes with an assertion error saying that the pixel buffer's pixel format and the metal texture's pixel format don't match.

[MTLDebugRenderCommandEncoder validateFramebufferWithRenderPipelineState:]:1288: failed assertion `Framebuffer With Render Pipeline State Validation

For color attachment 0, the render pipeline's pixelFormat (MTLPixelFormatRGBA16Float) does not match the framebuffer's pixelFormat (MTLPixelFormatBGRA8Unorm_sRGB).

For color attachment 1, the renderPipelineState pixelFormat must be MTLPixelFormatInvalid, as no texture is set.

So I tried setting the Metal Texture's pixel format like this

Code Block let pixelFormat = MTLPixelFormat.rgba16Uint

But now I'm getting another error I don't understand

_mtlValidateStrideTextureParameters:1656: failed assertion `Texture Descriptor Validation
IOSurface texture: bytesPerRow (5120) must be greater or equal to (10240) bytes

1.9k

Posted by

mortenjust

Reply

Add a Comment

Answer 1

I just checked the project files...

It is as I stated in my first reply, you need to ensure the texture settings match, and then you will not get the error regarding the textures.

After you change to rgba16 in both places (such as kCVPixelFormatType_64RGBAHalf), you will then be confronted by another texture setting match problem for the texture in the second attachment. Since you are overriding the render pass descriptor for SceneKit, it would be logically assumed that you then have to create and setup that texture on your own. If you spend more time with Metal, these things will be more obvious as they are very normal things.

I will leave that to you to complete, but I will say that the entire thing is clunky, extremely inefficient, and I would not personally recommended following this sample code's strategy or even using SceneKit or Swift. The use of AVAssetWriter is also not recommended. These, in their current form today and in the past, are Apple's half-baked attempts at providing convenience with a heavy price. There is little expectation based on Apple's prior results, that the standards of quality will change anytime soon. (Even though I hope for that every day)

I would instead recommend avoiding as many Apple frameworks as you can, and becoming more familiar with Metal and C, and get to your goals that way.

Posted by

MoreLightning

Add a Comment

Answer 2

It's telling you that your texture descriptor settings are incorrect relative to your other settings.

Notice how the number is exactly 1/2. Based on the limited information you wrote, it is expected because you went from BGRA8 to RGBA16, which is twice the size.

Posted by

MoreLightning

Add a Comment

Answer 3

Update: Thanks for the note, @MoreLightning. If you're curious, there was a link to my repository reproducing this behavior in the post. Here it is.

I've been at this all day. No solution. Here's what I tried

Attempt: Change the texture descriptor

Since I'm creating my Metal texture from a CVPixelbuffer with CVMetalTextureCacheCreateTextureFromImage, I can't figure out how to set its attributes - including the pixel format.

Attempt: Change the render pipeline's pixel format

I created a MTLRenderPipelineState and set the pixel format on that to match, but it was ignored. I think it's because of CVMetalTextureCacheCreateTextureFromImage

Attempt: Try H264

Didn't change anything. Also tried changing just the alpha quality, with HEVC with alpha, but no change.

Backing up a bit, I thought maybe isJitteringEnabled is not the way to get rid of jagged edges.

Attempt: Enable multi sampling

I was able to get my pipeline to pick up that I wanted multi sampling, but it crashes due to the texture not being set up for multisampling (more precisely a MTLTexture of type [.2DMultisample] (docs)

And then we're back at the problem of not being able to set the attributes for the MTLTexture.

Attempt: Copy the MTLTexture created by Core Video

I tried to use a MTLBlitCommandEncoder to copy the texture I was given by Core Video into a texture I had set up with the right attributes. But it crashes telling me that the attributes don't match.

I'm starting to think there's no solution to this?

Posted by

mortenjust

Add a Comment

Answer 4

I missed your link earlier...

I'll take a look tonight, did you first double check that isJitteringEnabled is in fact working as of today in a scenekit project without the rest of that sample project?

SceneKit is notorious for having bugs. So you should confirm that first. Then you should try changing Scenekit to rgba16 without the rest of that sample project. If you run into the same error, you'll know that you still need to change the texture settings.

Posted by

MoreLightning

Add a Comment

Answer 5

I just checked the project files...

It is as I stated in my first reply, you need to ensure the texture settings match, and then you will not get the error regarding the textures.

After you change to rgba16 in both places (such as kCVPixelFormatType_64RGBAHalf), you will then be confronted by another texture setting match problem for the texture in the second attachment. Since you are overriding the render pass descriptor for SceneKit, it would be logically assumed that you then have to create and setup that texture on your own. If you spend more time with Metal, these things will be more obvious as they are very normal things.

I will leave that to you to complete, but I will say that the entire thing is clunky, extremely inefficient, and I would not personally recommended following this sample code's strategy or even using SceneKit or Swift. The use of AVAssetWriter is also not recommended. These, in their current form today and in the past, are Apple's half-baked attempts at providing convenience with a heavy price. There is little expectation based on Apple's prior results, that the standards of quality will change anytime soon. (Even though I hope for that every day)

I would instead recommend avoiding as many Apple frameworks as you can, and becoming more familiar with Metal and C, and get to your goals that way.

Posted by

MoreLightning

Add a Comment

Answer 6

For people wondering how to actually set up the multipass texture
https://stackoverflow.com/questions/64958392/antialiasing-a-scenekit-rendering-with-metal

Posted by

mortenjust

Add a Comment

Crash when enabling jittering on SCNRenderer

Accepted Reply

Replies