Hi,
i have a SceneKit/ARKit (with my own metal shader) app and I need to change the main texture format from .bgra8unorm_srgb to .bgra8unorm. Since it is srgb, anything I draw is again converted to srgb and shown too bright.
Is there a way to change it/how can it be changed? I didn't find anything useful in docs or examples.
Thx
For everyone who also has this Problem (capturing ARKit video image and using it in your frag shader turning out too bright):
(This example is a part of my IOS app, so aspect ratio and such could/should change for other devices)
The conversion vom YCbCr_srgb to RGB conversion is a 2 step operation.
First you have to matrix-multiply your color with this YCbCr conversion matrix.
constant const float4x4 ycbcrToRGBTransform = float4x4(
+1.0000f, +1.0000f, +1.0000f, +0.0000f,
+0.0000f, -0.3441f, +1.7720f, +0.0000f,
+1.4020f, -0.7141f, +0.0000f, +0.0000f,
-0.7010f, +0.5291f, -0.8860f, +1.0000f
);
The second step is to adjust your gamma.
So you end up with something like this:
fragment float4 myFragmentShader(fragmentInOut in [[ stage_in ]],
texture2d<float, access::sample> capturedImageY [[ texture(0) ]],
texture2d<float, access::sample> capturedImageCBCR [[ texture(1) ]]) {
constexpr sampler s(mip_filter::linear,
mag_filter::linear,
min_filter::linear);
// in.displayBounds and in.position is calculated in the vertex shader, 0.615764 is aspect ratio of the video image size
float fx = (((in.displayBounds.x - in.position.x) / in.displayBounds.x) - 0.5) * 0.615764 + 0.5;
float fy = (((in.position.y) / in.displayBounds.y) - 0.5) * 1.0 + 0.5;
// Flip them around, since we're in portrait mode here, for full device orientation awareness, you have to add a bit more logic
const float2 tp = float2(fy, fx);
// Sample the color from the captured video frame
const float4 ycbcr = float4(capturedImageY.sample(s, tp).r, capturedImageCBCR.sample(s, tp).rg, 1.0);
// apply the YCbCr to RGB transform matrix and apply the gamma correction
const float3 rgbColor = pow((ycbcrToRGBTransform * ycbcr).rgb, 2.2);
// Now if you return this color, it will look exactly the same as the video background
return float4(rgbColor, 1.0);
}
This is answer close, but is not precisely correct so I'd expect your colors to be off a little bit in some cases. You should reference section 7.7.7 of the MSL spec (https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf) to see the exact conversion rule that Metal is applying when rendering to the sRGB drawable.
Hello AlgoChris,
Please request technical support for this question, and I can explain the options that you have to solve this problem!
I finally fixed it. My problem was the srgb gamma part. After I added a pow(capturedARImageColor.rgb,2.2) in my frag shader it works as intended now. But the documentation team really could do some examples specially with ARKit/SceneKit interactions, it is a rabbit hole with no exit.
Also the normal documentation pages are quite bare. Microsoft (as much as I dislike their company policies) did a very good job with theirs, they provide for everything a small example.
For example the attributes of vertex descriptors. To find out that they are already prefilled arrays (I still don't know how many entries they have, didn't bother to check) you have to dive deep.
Such things are quite unusual, but you still don't point it out in your documentation. Here, a small example would have saved me approx 1 hour, because I would've seen you just accessing the array entries without creating the objects. At least it would have made me think, why do you do it that way and I would have examined this particularly.
For everyone who also has this Problem (capturing ARKit video image and using it in your frag shader turning out too bright):
(This example is a part of my IOS app, so aspect ratio and such could/should change for other devices)
The conversion vom YCbCr_srgb to RGB conversion is a 2 step operation.
First you have to matrix-multiply your color with this YCbCr conversion matrix.
constant const float4x4 ycbcrToRGBTransform = float4x4(
+1.0000f, +1.0000f, +1.0000f, +0.0000f,
+0.0000f, -0.3441f, +1.7720f, +0.0000f,
+1.4020f, -0.7141f, +0.0000f, +0.0000f,
-0.7010f, +0.5291f, -0.8860f, +1.0000f
);
The second step is to adjust your gamma.
So you end up with something like this:
fragment float4 myFragmentShader(fragmentInOut in [[ stage_in ]],
texture2d<float, access::sample> capturedImageY [[ texture(0) ]],
texture2d<float, access::sample> capturedImageCBCR [[ texture(1) ]]) {
constexpr sampler s(mip_filter::linear,
mag_filter::linear,
min_filter::linear);
// in.displayBounds and in.position is calculated in the vertex shader, 0.615764 is aspect ratio of the video image size
float fx = (((in.displayBounds.x - in.position.x) / in.displayBounds.x) - 0.5) * 0.615764 + 0.5;
float fy = (((in.position.y) / in.displayBounds.y) - 0.5) * 1.0 + 0.5;
// Flip them around, since we're in portrait mode here, for full device orientation awareness, you have to add a bit more logic
const float2 tp = float2(fy, fx);
// Sample the color from the captured video frame
const float4 ycbcr = float4(capturedImageY.sample(s, tp).r, capturedImageCBCR.sample(s, tp).rg, 1.0);
// apply the YCbCr to RGB transform matrix and apply the gamma correction
const float3 rgbColor = pow((ycbcrToRGBTransform * ycbcr).rgb, 2.2);
// Now if you return this color, it will look exactly the same as the video background
return float4(rgbColor, 1.0);
}
This is answer close, but is not precisely correct so I'd expect your colors to be off a little bit in some cases. You should reference section 7.7.7 of the MSL spec (https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf) to see the exact conversion rule that Metal is applying when rendering to the sRGB drawable.
-
—
AlgoChris
Add a CommentOr what could help me too is a conversion matrix for YCbCr (srgb) to RGB (not SRGB). Or a conversion from SRGB to RGB.