ARKit video YCbCr -> RGB conversion matrix

Question

Created Oct ’21

Replies 5

Boosts 0

Views 1.3k

Participants 3

Hi all,

though I have the YCbCr texture of the camera feed, I still lack the right matrix to convert it to what ARKit is showing as background texture.

I tried it with this matrix (found here):

constant const float4x4 ycbcrToRGBTransform = float4x4(
    float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),
    float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),
    float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),
    float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f)
);

But this one is too bright.

Are there docs about this or is the background_video_frag source available? (Xcode tells me it's not/needs to be loaded, but I don't know from where)

I tried to find out from the buffers used in background_video_frag, but without the source it's hard to tell what's what.

Boost

Answer 1

Graphics and Games Engineer OP

Apple

Oct ’21

Hi AlgoChris,

Your color conversion matrix looks correct. Is is possible that the background texture (drawable) is using MTLPixelFormatBGRA8Unorm_sRGB? If so, can you try changing it to MTLPixelFormatBGRA8Unorm? The fact that your resulting images is too bright is most likely be caused by linear->sRGB conversion being applied twice.

0

Answer 2

AlgoChris OP

Oct ’21

Accepted Answer

Spot on, thanks a lot :)

import ARKit

public class MyARScnView : ARSCNView {
    public override var colorPixelFormat: MTLPixelFormat {
        get {
            return MTLPixelFormat.bgra8Unorm
        }
    }
}

fixed my issue

2

Answer 3

AlgoChris OP

Oct ’21

Ok, seems my answer didn't fix this problem at all. It is still too bright.

Which texture do you refer to when you talk about the background texture? Color 0? (I use SceneKit/SCNProgram to get my stuff on screen)

0

Answer 4

AlgoChris OP

Oct ’21

Ok, I think my question should be:

How do I change the color space in a SwiftUI app (IOS)?

0

Answer 5

DTS Engineer OP

Apple

Oct ’21

I don't think SwiftUI is the culprit here.

Technically, you could use the SCNDisableLinearSpaceRendering info.plist key, but if you do this it is likely that the rest of your scene content will render "incorrectly".

Alternatively, you can convert your sRGB camera image to linear in your shader, and then Metal will do its automatic linear -> sRGB conversion and everything should look "correct".

Please Request Technical Support if you need further assistance with this issue!

0