MetalKit

RSS for tag

Render graphics in a standard Metal view, load textures from many sources, and work efficiently with models provided by Model I/O using MetalKit.

Posts under MetalKit tag

200 Posts

Post

Replies

Boosts

Views

Activity

Metal Text Rendering w/ SF Fonts: What's Allowed?
I'm trying to render text with Metal for my (2D) game. I'm using the system fonts, e.g. the SF Pro family for English texts. I render the glyphs onto an atlas texture, and then sample from this texture. My questions: I assume that, for copyright reasons, I'm not allowed to include a pre-rendered font atlas in my app. Is my assumption correct? I can, however, have the app generate the atlas when it's first opened, and then use it within the app, right? If 2. is true, then can the app save the atlas somewhere in the app's private storage, so that it would not need to re-generate the atlas the next time? Thanks!
1
0
1.8k
Mar ’22
What handles interactions with Metal rendered images on macOS?
I have started learning Metal and the source text I was reading mentioned that interactions with images rendered to the screen is handled by Core Animation layers (CAMetalLayer) in iOS, but had no mention for any other platform. I am guessing for the other "mobile" platforms the same thing applies as iOS, but what handles this interaction for macOS, or is it also the same?
1
0
1.1k
Feb ’22
LiDAR Scanning in Realtime environment export object with realtime textures
I working on this project last 1 month I have gone through all the possibilities but have not found the expected output. I have the go through the following article and link: https://ixtli.unam.mx/6280-2/ https://developer.apple.com/documentation/arkit/content_anchors/scanning_and_detecting_3d_objects?language=objc https://github.com/TokyoYoshida/ExampleOfiOSLiDAR https://stackoverflow.com/questions/61063571/arkit-3-5-how-to-export-obj-from-new-ipad-pro-with-lidar/61104855#61104855 I am getting normal 3d objects with gray color, I am expecting objects with texture, if you guys know some way to find the solution then please let me know. Thank You!
0
0
1.8k
Jan ’22
Graphic Code Porting CGraphic to Metal
Hello Everybody. I'm trying to port graphic code written cg in unity to metal. And, one more thing I don't want to manually implement scene graph, so I gonna use SceneKit. So I should use SCNProgram or SCNNodeRendererDelegate, and I think SCNProgram is more comfort. And real my question is how convert this code, in cg Cull Front ZTest LEqual ZWrite On Blend SrcAlpha OneMinusSrcAlpha I know source alpha blending in MTLPipelineDescriptor, zbuffer in RenderCommandEncoder and Cull Face also. But When I use SCNProgram or SCNSceneRendererNode, can't find these options... how I change these. Help me.
0
0
894
Jan ’22
How to pass right model view projection matrix from scenekit to metal shader?
Hello, Everyone. I try to use a metal kit with a scene kit. Because, the scene kits scene graph is great, I want to implement a low-level metal shader. I want to use SCNNodeRenderDelegate, without SCNProgram. Because I want low-level implement for example passing custom extra MTLBuffer, or multi-pass-rendering. So I pass model view projection matrix like that, in metal shader struct NodeBuffer {   float4x4 modelTransform;   float4x4 modelViewProjectionTransform;   float4x4 modelViewTransform;   float4x4 normalTransform;   float2x3 boundingBox; }; in Swift code struct NodeMatrix: sizeable {     var modelTransform = float4x4()     var modelViewProjectionTransform = float4x4()     var modelViewTransform = float4x4()     var normalTransform = float4x4()     var boundingBox = float2x3()   } ...    private func updateNodeMatrix(_ camNode: SCNNode) {     guard let camera = camNode.camera else {       return     }           let modelMatrix = transform     let viewMatrix = camNode.transform     let projectionMatrix = camera.projectionTransform           let viewProjection = SCNMatrix4Mult(viewMatrix, projectionMatrix)     let modelViewProjection = SCNMatrix4Mult(modelMatrix, viewProjection)     nodeMatrix.modelViewProjectionTransform = float4x4(modelViewProjectionMatrix)   } ... public func renderNode(_ node: SCNNode,               renderer: SCNRenderer,               arguments: [String: Any])   {     guard let renderTexturePipelineState = renderTexturePipelineState,        let renderCommandEncoder = renderer.currentRenderCommandEncoder,        let camNode = renderer.pointOfView,        let texture = texture     else { return }           updateNodeMatrix(camNode)     guard let nodeBuffer       = renderer.device?.makeBuffer(bytes: &nodeMatrix,                      length: NodeMatrix.stride,                      options: [])     else { return }           renderCommandEncoder.setDepthStencilState(depthState)     renderCommandEncoder.setRenderPipelineState(renderTexturePipelineState)     renderCommandEncoder.setFragmentTexture(texture, index: 0)     renderCommandEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)     renderCommandEncoder.setVertexBuffer(nodeBuffer, offset: 0, index: 1)     renderCommandEncoder.drawIndexedPrimitives(type: .triangle,                           indexCount: indexCount,                           indexType: .uint16,                           indexBuffer: indexBuffer,                           indexBufferOffset: 0)   } But I got the wrong model view projection matrix in the shader. I think scene kit has modify intermediate transform hiding. I can't know, help me...
1
0
2.1k
Jan ’22
MTLDebugCommandBuffer lockPurgeableObjects
I'm using Xcode 13 after recently updating to MacOS Monterey, and only after updating am I getting this error: [MTLDebugCommandBuffer lockPurgeableObjects]:2103: failed assertion `MTLResource 0x14a8a8cc0 (label: null), referenced in cmd buffer 0x149091400 (label: null) is in volatile or empty purgeable state at commit' I haven't changed my code at all between updating to the latest OS, and it worked perfectly before. How can I fix this? I don't think there should be any reason that I can't use a command buffer on a texture resource with a volatile/empty purgeable state.
4
0
3.9k
Jan ’22
Pass classifications from Reconstructed Scene Understanding to Metal renderer
Hey everyone :-) I'm trying access the information of my ARFrame so I can colorize different parts of my scene depending of their classifications. In other words, I'm looking for a way to transform XYZ World Coordinates or XY Screen Coordinates to what is being classified by my AR Session in that position. All the floor in the scene being colorize in red, the tables in blue, the walls in green, etc. How could I embed in real-time these classifications from my reconstructed ARMeshAnchor into my coordinates so I can access it on my Metal Script in real-time? As of today... I've been able to get the XYZ World Coordinate related to a XY Screen Coordinate reading the following example: Building an Immersive Experience with RealityKit Here is a scene painting it with different shades of purple depending on the Z world-coordinate: P.S. I want to be able to do it within RealityKit. Thanks in advance.
2
0
865
Jan ’22
RealityKit AR Depth Put The Texture Exclusively On The Floor
Hello everyone! how to put a custom shader in depth only on the floor? I'm trying to use the depth of the scene to put the shader exclusively on the floor, but apparently I'm doing something wrong Links https://www.dropbox.com/s/4ghun92frlcg7hz/IMG_9960.PNG?dl=0 https://www.dropbox.com/home?preview=-2362988581602429186.MP4 PostProcess.metal float linearizeDepth(float sampleDepth, float4x4 viewMatrix) {     constexpr float kDepthEpsilon = 1e-5f;     float d = max(kDepthEpsilon, sampleDepth);     d = abs(-viewMatrix[3][2] / d);     return d; } constexpr sampler textureSampler(address::clamp_to_edge, filter::bicubic); float getDepth(float2 coords, constant InputArgs *args, texture2d<float, access::sample> inDepth,depth2d<float, access::sample> arDepth) {     float2 arDepthCoords = args->orientationTransform * coords + args->orientationOffset;     float realDepth = arDepth.sample(textureSampler, arDepthCoords);     float virtualDepth = linearizeDepth(inDepth.sample(textureSampler, coords)[0], args->viewMatrix);     bool realFragment = (virtualDepth <= FLT_EPSILON);     if (realFragment) { virtualDepth = realDepth; }     return min(virtualDepth, realDepth); } float3 getDirection(float2 screenCoord, constant InputArgs *args) {     float3 top = mix(args->topLeft.xyz, args->topRight.xyz, screenCoord.x);     float3 bottom = mix(args->bottomLeft.xyz, args->bottomRight.xyz, screenCoord.x);     return normalize(mix(bottom, top, screenCoord.y)); } float3 worldCoordsForDepth(float depth, float2 screenCords, constant InputArgs *args) {     float3 centerDirection = getDirection(float2(0.5, 0.5), args);     float3 direction = getDirection(screenCords, args);     float depth2 = depth / dot(direction, centerDirection);     return direction * depth2 + args->viewTranslation.xyz; } [[kernel]] void postProcess(uint2 gid [[thread_position_in_grid]],                  texture2d<half, access::read> inputTexture [[texture(0)]],                  texture2d<float, access::sample> inDepth [[texture(1)]],                  texture2d<half, access::write> outputTexture [[texture(2)]],                  depth2d<float, access::sample> arDepth [[texture(3)]],                  constant InputArgs *args [[buffer(0)]]) {  float2 screenCoords = float2(float(gid[0]) / float(outputTexture.get_width()),                                  float(gid[1]) / float(outputTexture.get_height()));     float rawDepth = getDepth(screenCoords, args, inDepth, arDepth);     float3 worldCoords = worldCoordsForDepth(rawDepth, screenCoords, args);     float depth = rawDepth;     depth = 1 - pow(1 / (pow(depth, args->intensity) + 1), args->falloff);     depth = clamp(depth, 0.0, 1.0);     half4 nearColor = inputTexture.read(gid);     float blend = pow(1 - depth, args->exponent);     half4 color = half4(0.0);  float2 frag = worldCoords.xz;         frag *= 1.0 - 0.2 * cos (frag) * sin (3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.0)).x);         frag *= 5.0;         float random = rand (floor(frag));         float2 black = smoothstep(1.0, 0.8, cos(frag * 3.14159 * 2.0));         float3 finalColor = hsv2rgb (float3 (random, 1.0, 1.0));         finalColor *= black.x * black.y * smoothstep (1.0, 0.0, length (fract (frag) - 0.5));         finalColor *= 0.5 + 0.5 * cos (random + random * args->time + args->time + 3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.7)).x); color = blend * nearColor + (1.0 - blend) * half4(half3(finalColor), 1.0); } I really hope for help of understanding this one
1
0
1.1k
Dec ’21
How to use the "write" function in a CoreImage Metal shader
There is a write function documented in the CoreImage Metal shader reference here: https://developer.apple.com/metal/MetalCIKLReference6.pdf But I'm not sure how to use it. I assumed one would be able to use it on the destination parameter i.e. dest.write(...) but I get the error, "no member named 'write' in 'coreimage::destination'" How do I use this function?
1
0
1.2k
Dec ’21
Custom CoreImage Box Blur kernel Much slower than equivalent built-in CIBoxBlur
I've created a custom BoxBlur kernel that produces identical results to Apple's built-in box blur (CIBoxBlur) kernel but my custom kernel is orders of magnitude slower. So naturally I am wondering what I'm doing wrong to get such poor performance. Below is my custom kernel in the Metal shading language. Can you spot why it's so slow? The built-in filter performs well so I can only assume it's something I'm doing wrong. #include <CoreImage/CoreImage.h> #import <simd/simd.h> extern "C" { namespace coreimage { float4 customBoxBlurFilterKernel(sampler src) { float2 crd = src.coord(); int edge = 100; int minx = crd.x - edge; int maxx = crd.x + edge; int miny = crd.y - edge; int maxy = crd.y + edge; float4 sums = float4(0,0,0,0); float cnt = 0; // compute average of surrounding rgb values for(int row=miny; row < maxy; row++) { for(int col=minx; col < maxx; col++) { float4 samp = src.sample(float2(col, row)); sums[0] += samp[0]; sums[1] += samp[1]; sums[2] += samp[2]; cnt += 1.; } } return float4(sums[0]/cnt, sums[1]/cnt, sums[2]/cnt, 1); } } }
2
0
1.7k
Dec ’21
What is the color space of the environment texture of AREnvironmentProbeAnchor?
The documentation says the pixel format of the environmentTexture in an AREnvironmentProbeAnchor is bgra8Unorm_srgb. https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor/2977511-environmenttexture However, when I inspect the pixelFormat property of the MTLTexture it says it's rgba16Float. I'm trying to read the texture out as a PNG, and because it's a 16-bit float image, I'm assuming its color space is CGColorSpace.displayP3, but I'm not 100% sure. The texture looks darker than what I expected. Could it be that the color space is sRGB, but it's 16-bit because it's actually an HDR texture stored as linear RGB? (Tested on iPhone 12, iOS 15)
2
0
1.2k
Dec ’21
Get camera pose information for point cloud calibration
I’m trying to acquire a point cloud and then capture an image (the capture would stop the point cloud acquisition session). The point cloud acquisition seems easy. This is a well discussed topic: https://developer.apple.com/forums/thread/658109 Furthermore, it would seem possible to store camera pose information from the point cloud with the photo so that data could be used to find a pose. Later, I’d align the 3D point cloud to that 2d image (preferably not using depth images, interpolation, or NNs). I am surprised, but seems really hard to find out the feasibility of such an approach.
0
0
708
Dec ’21
Multiple object rendering in Metalkit
I am working on rendering multiple objects in Metalkit. While doing it I can see objects are flickering if they are overlaying each-other. I'm passing the currentRenderPassDescriptor and current drawable in draw method which is calling every time. Please refer screenshot. Please help me out to render objects without flickering.
3
0
734
Nov ’21
Proper MTKView MTLPixelFormat for P3 Mac screens
For running my app on devices with P3 screens, to take advantage of the P3 colors, which MTLPixelFormat is the proper one to use for my MTKView? My app is for iOS, but it’s nice to be able to debug on the simulator or as a Catalyst app. Right now, I’ve been using MTLPixelFormatBGR10_XR_sRGB and MTLPixelFormatBGRA10_XR_sRGB per the recommendation in WWDC 2016 session 605 for iOS support of P3 color. This appears to work fine on iOS devices, but on my Apple silicon Mac, when running as a Catalyst app, the colors sometimes look lighter than they should, but it doesn’t always happen. When I run in the simulator on this Mac, my app crashes. When I was using an Intel Mac (with an sRGB screen), I was having my app render in sRGB-only with MTLPixelFormat.bgra8Unorm. MTLPixelFormat.bgra8Unorm also works fine on my Apple silicon Mac, but of course, I’m not getting the P3 color space.  For Apple Silicon Macs, as the GPU hardware is more like an iOS device than like an Intel Mac, should the same pixel formats work on an Apple silicon Mac as on iOS devices? Or are the pixel formats that work with Apple Silicon Macs the same as those on Intel Macs?
1
0
901
Nov ’21
what "Zero Metal services found" means ?
Friendly greetings ! I'm on a MacBook Air M1, using Xcode 13.1, Swift. The full source code is below, as a MaOS Console Application import Foundation import Metal import MetalKit import simd guard let device : MTLDevice = MTLCreateSystemDefaultDevice() else { fatalError("could not create metal default device") } let queue = device.makeCommandQueue() print("Hello, World!") The full output, running in debug from XCode is : 2021-10-29 06:00:16.567089+0200 FraCompute[71702:11572598] Metal API Validation Enabled 2021-10-29 06:00:16.621413+0200 FraCompute[71702:11572598] flock failed to lock list file (/var/folders/s7/7m1rt8kx3jq7c02mwclmnvf80000gn/C//com.apple.metal/16777235_322/functions.list): errno = 35 2021-10-29 06:00:16.628053+0200 FraCompute[71702:11572598] +[MTLIOAccelDevice registerDevices]: Zero Metal services found Hello, World! Program ended with exit code: 0 Running the sample "MacOS Game" default code (the one with the rotating cube) also give this "Zero Metal services found". What's a "metal service" anyway ? Googling for it obviously give me unrelated smithing service offer =^_^= It's not even "Metal Service not found" but, "I found Zero Metal Services", whatever that's supposed to mean ? is it "I didn't find any metal service" or is there a thing called "Zero Metal" ? I can query the metal device name and it get me "Apple M1". No problem, everything seems to work fine (so far). But what's the meaning of this message please ?
1
0
1.9k
Oct ’21
ARKit video YCbCr -> RGB conversion matrix
Hi all, though I have the YCbCr texture of the camera feed, I still lack the right matrix to convert it to what ARKit is showing as background texture. I tried it with this matrix (found here): constant const float4x4 ycbcrToRGBTransform = float4x4(     float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),     float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),     float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),     float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f) ); But this one is too bright. Are there docs about this or is the background_video_frag source available? (Xcode tells me it's not/needs to be loaded, but I don't know from where) I tried to find out from the buffers used in background_video_frag, but without the source it's hard to tell what's what.
5
0
1.4k
Oct ’21
Render MTKView with different Colorspace
Is it possible to render an MTKView in different color spaces? For example, if I want to render it in CMYK, is there a way to adjust the colors on every frame to that colorspace before presenting it on the screen, or is that something that needs to be handled in a shader?
Replies
3
Boosts
0
Views
2.2k
Activity
Mar ’22
Metal Text Rendering w/ SF Fonts: What's Allowed?
I'm trying to render text with Metal for my (2D) game. I'm using the system fonts, e.g. the SF Pro family for English texts. I render the glyphs onto an atlas texture, and then sample from this texture. My questions: I assume that, for copyright reasons, I'm not allowed to include a pre-rendered font atlas in my app. Is my assumption correct? I can, however, have the app generate the atlas when it's first opened, and then use it within the app, right? If 2. is true, then can the app save the atlas somewhere in the app's private storage, so that it would not need to re-generate the atlas the next time? Thanks!
Replies
1
Boosts
0
Views
1.8k
Activity
Mar ’22
What handles interactions with Metal rendered images on macOS?
I have started learning Metal and the source text I was reading mentioned that interactions with images rendered to the screen is handled by Core Animation layers (CAMetalLayer) in iOS, but had no mention for any other platform. I am guessing for the other "mobile" platforms the same thing applies as iOS, but what handles this interaction for macOS, or is it also the same?
Replies
1
Boosts
0
Views
1.1k
Activity
Feb ’22
Passing MTLTexture to Metal Core Image Kernel
Is it possible to pass MTLTexture to Metal Core Image Kernel? How can Metal resources be shared with Core Image?
Replies
1
Boosts
0
Views
1.4k
Activity
Feb ’22
LiDAR Scanning in Realtime environment export object with realtime textures
I working on this project last 1 month I have gone through all the possibilities but have not found the expected output. I have the go through the following article and link: https://ixtli.unam.mx/6280-2/ https://developer.apple.com/documentation/arkit/content_anchors/scanning_and_detecting_3d_objects?language=objc https://github.com/TokyoYoshida/ExampleOfiOSLiDAR https://stackoverflow.com/questions/61063571/arkit-3-5-how-to-export-obj-from-new-ipad-pro-with-lidar/61104855#61104855 I am getting normal 3d objects with gray color, I am expecting objects with texture, if you guys know some way to find the solution then please let me know. Thank You!
Replies
0
Boosts
0
Views
1.8k
Activity
Jan ’22
Graphic Code Porting CGraphic to Metal
Hello Everybody. I'm trying to port graphic code written cg in unity to metal. And, one more thing I don't want to manually implement scene graph, so I gonna use SceneKit. So I should use SCNProgram or SCNNodeRendererDelegate, and I think SCNProgram is more comfort. And real my question is how convert this code, in cg Cull Front ZTest LEqual ZWrite On Blend SrcAlpha OneMinusSrcAlpha I know source alpha blending in MTLPipelineDescriptor, zbuffer in RenderCommandEncoder and Cull Face also. But When I use SCNProgram or SCNSceneRendererNode, can't find these options... how I change these. Help me.
Replies
0
Boosts
0
Views
894
Activity
Jan ’22
How to pass right model view projection matrix from scenekit to metal shader?
Hello, Everyone. I try to use a metal kit with a scene kit. Because, the scene kits scene graph is great, I want to implement a low-level metal shader. I want to use SCNNodeRenderDelegate, without SCNProgram. Because I want low-level implement for example passing custom extra MTLBuffer, or multi-pass-rendering. So I pass model view projection matrix like that, in metal shader struct NodeBuffer {   float4x4 modelTransform;   float4x4 modelViewProjectionTransform;   float4x4 modelViewTransform;   float4x4 normalTransform;   float2x3 boundingBox; }; in Swift code struct NodeMatrix: sizeable {     var modelTransform = float4x4()     var modelViewProjectionTransform = float4x4()     var modelViewTransform = float4x4()     var normalTransform = float4x4()     var boundingBox = float2x3()   } ...    private func updateNodeMatrix(_ camNode: SCNNode) {     guard let camera = camNode.camera else {       return     }           let modelMatrix = transform     let viewMatrix = camNode.transform     let projectionMatrix = camera.projectionTransform           let viewProjection = SCNMatrix4Mult(viewMatrix, projectionMatrix)     let modelViewProjection = SCNMatrix4Mult(modelMatrix, viewProjection)     nodeMatrix.modelViewProjectionTransform = float4x4(modelViewProjectionMatrix)   } ... public func renderNode(_ node: SCNNode,               renderer: SCNRenderer,               arguments: [String: Any])   {     guard let renderTexturePipelineState = renderTexturePipelineState,        let renderCommandEncoder = renderer.currentRenderCommandEncoder,        let camNode = renderer.pointOfView,        let texture = texture     else { return }           updateNodeMatrix(camNode)     guard let nodeBuffer       = renderer.device?.makeBuffer(bytes: &nodeMatrix,                      length: NodeMatrix.stride,                      options: [])     else { return }           renderCommandEncoder.setDepthStencilState(depthState)     renderCommandEncoder.setRenderPipelineState(renderTexturePipelineState)     renderCommandEncoder.setFragmentTexture(texture, index: 0)     renderCommandEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)     renderCommandEncoder.setVertexBuffer(nodeBuffer, offset: 0, index: 1)     renderCommandEncoder.drawIndexedPrimitives(type: .triangle,                           indexCount: indexCount,                           indexType: .uint16,                           indexBuffer: indexBuffer,                           indexBufferOffset: 0)   } But I got the wrong model view projection matrix in the shader. I think scene kit has modify intermediate transform hiding. I can't know, help me...
Replies
1
Boosts
0
Views
2.1k
Activity
Jan ’22
MTLDebugCommandBuffer lockPurgeableObjects
I'm using Xcode 13 after recently updating to MacOS Monterey, and only after updating am I getting this error: [MTLDebugCommandBuffer lockPurgeableObjects]:2103: failed assertion `MTLResource 0x14a8a8cc0 (label: null), referenced in cmd buffer 0x149091400 (label: null) is in volatile or empty purgeable state at commit' I haven't changed my code at all between updating to the latest OS, and it worked perfectly before. How can I fix this? I don't think there should be any reason that I can't use a command buffer on a texture resource with a volatile/empty purgeable state.
Replies
4
Boosts
0
Views
3.9k
Activity
Jan ’22
com.metal.completionqueuedispatch EXC_BAD_ACCESS crash in Xcode
I am getting this error while running the app with Xcode. especially when I am moving things in view like swiping with my finger, rotating the screen etc.. I have not been able to reproduce it when running the app like normal user. I am just wondering if Xcode is doing something weird in debugging and is hitting the apps animations.
Replies
3
Boosts
0
Views
2.0k
Activity
Jan ’22
Pass classifications from Reconstructed Scene Understanding to Metal renderer
Hey everyone :-) I'm trying access the information of my ARFrame so I can colorize different parts of my scene depending of their classifications. In other words, I'm looking for a way to transform XYZ World Coordinates or XY Screen Coordinates to what is being classified by my AR Session in that position. All the floor in the scene being colorize in red, the tables in blue, the walls in green, etc. How could I embed in real-time these classifications from my reconstructed ARMeshAnchor into my coordinates so I can access it on my Metal Script in real-time? As of today... I've been able to get the XYZ World Coordinate related to a XY Screen Coordinate reading the following example: Building an Immersive Experience with RealityKit Here is a scene painting it with different shades of purple depending on the Z world-coordinate: P.S. I want to be able to do it within RealityKit. Thanks in advance.
Replies
2
Boosts
0
Views
865
Activity
Jan ’22
RealityKit AR Depth Put The Texture Exclusively On The Floor
Hello everyone! how to put a custom shader in depth only on the floor? I'm trying to use the depth of the scene to put the shader exclusively on the floor, but apparently I'm doing something wrong Links https://www.dropbox.com/s/4ghun92frlcg7hz/IMG_9960.PNG?dl=0 https://www.dropbox.com/home?preview=-2362988581602429186.MP4 PostProcess.metal float linearizeDepth(float sampleDepth, float4x4 viewMatrix) {     constexpr float kDepthEpsilon = 1e-5f;     float d = max(kDepthEpsilon, sampleDepth);     d = abs(-viewMatrix[3][2] / d);     return d; } constexpr sampler textureSampler(address::clamp_to_edge, filter::bicubic); float getDepth(float2 coords, constant InputArgs *args, texture2d<float, access::sample> inDepth,depth2d<float, access::sample> arDepth) {     float2 arDepthCoords = args->orientationTransform * coords + args->orientationOffset;     float realDepth = arDepth.sample(textureSampler, arDepthCoords);     float virtualDepth = linearizeDepth(inDepth.sample(textureSampler, coords)[0], args->viewMatrix);     bool realFragment = (virtualDepth <= FLT_EPSILON);     if (realFragment) { virtualDepth = realDepth; }     return min(virtualDepth, realDepth); } float3 getDirection(float2 screenCoord, constant InputArgs *args) {     float3 top = mix(args->topLeft.xyz, args->topRight.xyz, screenCoord.x);     float3 bottom = mix(args->bottomLeft.xyz, args->bottomRight.xyz, screenCoord.x);     return normalize(mix(bottom, top, screenCoord.y)); } float3 worldCoordsForDepth(float depth, float2 screenCords, constant InputArgs *args) {     float3 centerDirection = getDirection(float2(0.5, 0.5), args);     float3 direction = getDirection(screenCords, args);     float depth2 = depth / dot(direction, centerDirection);     return direction * depth2 + args->viewTranslation.xyz; } [[kernel]] void postProcess(uint2 gid [[thread_position_in_grid]],                  texture2d<half, access::read> inputTexture [[texture(0)]],                  texture2d<float, access::sample> inDepth [[texture(1)]],                  texture2d<half, access::write> outputTexture [[texture(2)]],                  depth2d<float, access::sample> arDepth [[texture(3)]],                  constant InputArgs *args [[buffer(0)]]) {  float2 screenCoords = float2(float(gid[0]) / float(outputTexture.get_width()),                                  float(gid[1]) / float(outputTexture.get_height()));     float rawDepth = getDepth(screenCoords, args, inDepth, arDepth);     float3 worldCoords = worldCoordsForDepth(rawDepth, screenCoords, args);     float depth = rawDepth;     depth = 1 - pow(1 / (pow(depth, args->intensity) + 1), args->falloff);     depth = clamp(depth, 0.0, 1.0);     half4 nearColor = inputTexture.read(gid);     float blend = pow(1 - depth, args->exponent);     half4 color = half4(0.0);  float2 frag = worldCoords.xz;         frag *= 1.0 - 0.2 * cos (frag) * sin (3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.0)).x);         frag *= 5.0;         float random = rand (floor(frag));         float2 black = smoothstep(1.0, 0.8, cos(frag * 3.14159 * 2.0));         float3 finalColor = hsv2rgb (float3 (random, 1.0, 1.0));         finalColor *= black.x * black.y * smoothstep (1.0, 0.0, length (fract (frag) - 0.5));         finalColor *= 0.5 + 0.5 * cos (random + random * args->time + args->time + 3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.7)).x); color = blend * nearColor + (1.0 - blend) * half4(half3(finalColor), 1.0); } I really hope for help of understanding this one
Replies
1
Boosts
0
Views
1.1k
Activity
Dec ’21
How to use the "write" function in a CoreImage Metal shader
There is a write function documented in the CoreImage Metal shader reference here: https://developer.apple.com/metal/MetalCIKLReference6.pdf But I'm not sure how to use it. I assumed one would be able to use it on the destination parameter i.e. dest.write(...) but I get the error, "no member named 'write' in 'coreimage::destination'" How do I use this function?
Replies
1
Boosts
0
Views
1.2k
Activity
Dec ’21
Custom CoreImage Box Blur kernel Much slower than equivalent built-in CIBoxBlur
I've created a custom BoxBlur kernel that produces identical results to Apple's built-in box blur (CIBoxBlur) kernel but my custom kernel is orders of magnitude slower. So naturally I am wondering what I'm doing wrong to get such poor performance. Below is my custom kernel in the Metal shading language. Can you spot why it's so slow? The built-in filter performs well so I can only assume it's something I'm doing wrong. #include <CoreImage/CoreImage.h> #import <simd/simd.h> extern "C" { namespace coreimage { float4 customBoxBlurFilterKernel(sampler src) { float2 crd = src.coord(); int edge = 100; int minx = crd.x - edge; int maxx = crd.x + edge; int miny = crd.y - edge; int maxy = crd.y + edge; float4 sums = float4(0,0,0,0); float cnt = 0; // compute average of surrounding rgb values for(int row=miny; row < maxy; row++) { for(int col=minx; col < maxx; col++) { float4 samp = src.sample(float2(col, row)); sums[0] += samp[0]; sums[1] += samp[1]; sums[2] += samp[2]; cnt += 1.; } } return float4(sums[0]/cnt, sums[1]/cnt, sums[2]/cnt, 1); } } }
Replies
2
Boosts
0
Views
1.7k
Activity
Dec ’21
What is the color space of the environment texture of AREnvironmentProbeAnchor?
The documentation says the pixel format of the environmentTexture in an AREnvironmentProbeAnchor is bgra8Unorm_srgb. https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor/2977511-environmenttexture However, when I inspect the pixelFormat property of the MTLTexture it says it's rgba16Float. I'm trying to read the texture out as a PNG, and because it's a 16-bit float image, I'm assuming its color space is CGColorSpace.displayP3, but I'm not 100% sure. The texture looks darker than what I expected. Could it be that the color space is sRGB, but it's 16-bit because it's actually an HDR texture stored as linear RGB? (Tested on iPhone 12, iOS 15)
Replies
2
Boosts
0
Views
1.2k
Activity
Dec ’21
Get camera pose information for point cloud calibration
I’m trying to acquire a point cloud and then capture an image (the capture would stop the point cloud acquisition session). The point cloud acquisition seems easy. This is a well discussed topic: https://developer.apple.com/forums/thread/658109 Furthermore, it would seem possible to store camera pose information from the point cloud with the photo so that data could be used to find a pose. Later, I’d align the 3D point cloud to that 2d image (preferably not using depth images, interpolation, or NNs). I am surprised, but seems really hard to find out the feasibility of such an approach.
Replies
0
Boosts
0
Views
708
Activity
Dec ’21
Multiple object rendering in Metalkit
I am working on rendering multiple objects in Metalkit. While doing it I can see objects are flickering if they are overlaying each-other. I'm passing the currentRenderPassDescriptor and current drawable in draw method which is calling every time. Please refer screenshot. Please help me out to render objects without flickering.
Replies
3
Boosts
0
Views
734
Activity
Nov ’21
Proper MTKView MTLPixelFormat for P3 Mac screens
For running my app on devices with P3 screens, to take advantage of the P3 colors, which MTLPixelFormat is the proper one to use for my MTKView? My app is for iOS, but it’s nice to be able to debug on the simulator or as a Catalyst app. Right now, I’ve been using MTLPixelFormatBGR10_XR_sRGB and MTLPixelFormatBGRA10_XR_sRGB per the recommendation in WWDC 2016 session 605 for iOS support of P3 color. This appears to work fine on iOS devices, but on my Apple silicon Mac, when running as a Catalyst app, the colors sometimes look lighter than they should, but it doesn’t always happen. When I run in the simulator on this Mac, my app crashes. When I was using an Intel Mac (with an sRGB screen), I was having my app render in sRGB-only with MTLPixelFormat.bgra8Unorm. MTLPixelFormat.bgra8Unorm also works fine on my Apple silicon Mac, but of course, I’m not getting the P3 color space.  For Apple Silicon Macs, as the GPU hardware is more like an iOS device than like an Intel Mac, should the same pixel formats work on an Apple silicon Mac as on iOS devices? Or are the pixel formats that work with Apple Silicon Macs the same as those on Intel Macs?
Replies
1
Boosts
0
Views
901
Activity
Nov ’21
How to change SceneKit/ARKit app texture format?
Hi, i have a SceneKit/ARKit (with my own metal shader) app and I need to change the main texture format from .bgra8unorm_srgb to .bgra8unorm. Since it is srgb, anything I draw is again converted to srgb and shown too bright. Is there a way to change it/how can it be changed? I didn't find anything useful in docs or examples. Thx
Replies
3
Boosts
0
Views
1.3k
Activity
Nov ’21
what "Zero Metal services found" means ?
Friendly greetings ! I'm on a MacBook Air M1, using Xcode 13.1, Swift. The full source code is below, as a MaOS Console Application import Foundation import Metal import MetalKit import simd guard let device : MTLDevice = MTLCreateSystemDefaultDevice() else { fatalError("could not create metal default device") } let queue = device.makeCommandQueue() print("Hello, World!") The full output, running in debug from XCode is : 2021-10-29 06:00:16.567089+0200 FraCompute[71702:11572598] Metal API Validation Enabled 2021-10-29 06:00:16.621413+0200 FraCompute[71702:11572598] flock failed to lock list file (/var/folders/s7/7m1rt8kx3jq7c02mwclmnvf80000gn/C//com.apple.metal/16777235_322/functions.list): errno = 35 2021-10-29 06:00:16.628053+0200 FraCompute[71702:11572598] +[MTLIOAccelDevice registerDevices]: Zero Metal services found Hello, World! Program ended with exit code: 0 Running the sample "MacOS Game" default code (the one with the rotating cube) also give this "Zero Metal services found". What's a "metal service" anyway ? Googling for it obviously give me unrelated smithing service offer =^_^= It's not even "Metal Service not found" but, "I found Zero Metal Services", whatever that's supposed to mean ? is it "I didn't find any metal service" or is there a thing called "Zero Metal" ? I can query the metal device name and it get me "Apple M1". No problem, everything seems to work fine (so far). But what's the meaning of this message please ?
Replies
1
Boosts
0
Views
1.9k
Activity
Oct ’21
ARKit video YCbCr -> RGB conversion matrix
Hi all, though I have the YCbCr texture of the camera feed, I still lack the right matrix to convert it to what ARKit is showing as background texture. I tried it with this matrix (found here): constant const float4x4 ycbcrToRGBTransform = float4x4(     float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),     float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),     float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),     float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f) ); But this one is too bright. Are there docs about this or is the background_video_frag source available? (Xcode tells me it's not/needs to be loaded, but I don't know from where) I tried to find out from the buffers used in background_video_frag, but without the source it's hard to tell what's what.
Replies
5
Boosts
0
Views
1.4k
Activity
Oct ’21