Is it possible to pass MTLTexture to Metal Core Image Kernel? How can Metal resources be shared with Core Image?

I working on this project last 1 month I have gone through all the possibilities but have not found the expected output. I have the go through the following article and link: https://ixtli.unam.mx/6280-2/ https://developer.apple.com/documentation/arkit/content_anchors/scanning_and_detecting_3d_objects?language=objc https://github.com/TokyoYoshida/ExampleOfiOSLiDAR https://stackoverflow.com/questions/61063571/arkit-3-5-how-to-export-obj-from-new-ipad-pro-with-lidar/61104855#61104855 I am getting normal 3d objects with gray color, I am expecting objects with texture, if you guys know some way to find the solution then please let me know. Thank You!

Hello Everybody. I'm trying to port graphic code written cg in unity to metal. And, one more thing I don't want to manually implement scene graph, so I gonna use SceneKit. So I should use SCNProgram or SCNNodeRendererDelegate, and I think SCNProgram is more comfort. And real my question is how convert this code, in cg Cull Front ZTest LEqual ZWrite On Blend SrcAlpha OneMinusSrcAlpha I know source alpha blending in MTLPipelineDescriptor, zbuffer in RenderCommandEncoder and Cull Face also. But When I use SCNProgram or SCNSceneRendererNode, can't find these options... how I change these. Help me.

Hello, Everyone. I try to use a metal kit with a scene kit. Because, the scene kits scene graph is great, I want to implement a low-level metal shader. I want to use SCNNodeRenderDelegate, without SCNProgram. Because I want low-level implement for example passing custom extra MTLBuffer, or multi-pass-rendering. So I pass model view projection matrix like that, in metal shader struct NodeBuffer {   float4x4 modelTransform;   float4x4 modelViewProjectionTransform;   float4x4 modelViewTransform;   float4x4 normalTransform;   float2x3 boundingBox; }; in Swift code struct NodeMatrix: sizeable {     var modelTransform = float4x4()     var modelViewProjectionTransform = float4x4()     var modelViewTransform = float4x4()     var normalTransform = float4x4()     var boundingBox = float2x3()   } ...    private func updateNodeMatrix(_ camNode: SCNNode) {     guard let camera = camNode.camera else {       return     }           let modelMatrix = transform     let viewMatrix = camNode.transform     let projectionMatrix = camera.projectionTransform           let viewProjection = SCNMatrix4Mult(viewMatrix, projectionMatrix)     let modelViewProjection = SCNMatrix4Mult(modelMatrix, viewProjection)     nodeMatrix.modelViewProjectionTransform = float4x4(modelViewProjectionMatrix)   } ... public func renderNode(_ node: SCNNode,               renderer: SCNRenderer,               arguments: [String: Any])   {     guard let renderTexturePipelineState = renderTexturePipelineState,        let renderCommandEncoder = renderer.currentRenderCommandEncoder,        let camNode = renderer.pointOfView,        let texture = texture     else { return }           updateNodeMatrix(camNode)     guard let nodeBuffer       = renderer.device?.makeBuffer(bytes: &nodeMatrix,                      length: NodeMatrix.stride,                      options: [])     else { return }           renderCommandEncoder.setDepthStencilState(depthState)     renderCommandEncoder.setRenderPipelineState(renderTexturePipelineState)     renderCommandEncoder.setFragmentTexture(texture, index: 0)     renderCommandEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)     renderCommandEncoder.setVertexBuffer(nodeBuffer, offset: 0, index: 1)     renderCommandEncoder.drawIndexedPrimitives(type: .triangle,                           indexCount: indexCount,                           indexType: .uint16,                           indexBuffer: indexBuffer,                           indexBufferOffset: 0)   } But I got the wrong model view projection matrix in the shader. I think scene kit has modify intermediate transform hiding. I can't know, help me...

I'm using Xcode 13 after recently updating to MacOS Monterey, and only after updating am I getting this error: [MTLDebugCommandBuffer lockPurgeableObjects]:2103: failed assertion `MTLResource 0x14a8a8cc0 (label: null), referenced in cmd buffer 0x149091400 (label: null) is in volatile or empty purgeable state at commit' I haven't changed my code at all between updating to the latest OS, and it worked perfectly before. How can I fix this? I don't think there should be any reason that I can't use a command buffer on a texture resource with a volatile/empty purgeable state.

I am getting this error while running the app with Xcode. especially when I am moving things in view like swiping with my finger, rotating the screen etc.. I have not been able to reproduce it when running the app like normal user. I am just wondering if Xcode is doing something weird in debugging and is hitting the apps animations.

Hey everyone :-) I'm trying access the information of my ARFrame so I can colorize different parts of my scene depending of their classifications. In other words, I'm looking for a way to transform XYZ World Coordinates or XY Screen Coordinates to what is being classified by my AR Session in that position. All the floor in the scene being colorize in red, the tables in blue, the walls in green, etc. How could I embed in real-time these classifications from my reconstructed ARMeshAnchor into my coordinates so I can access it on my Metal Script in real-time? As of today... I've been able to get the XYZ World Coordinate related to a XY Screen Coordinate reading the following example: Building an Immersive Experience with RealityKit Here is a scene painting it with different shades of purple depending on the Z world-coordinate: P.S. I want to be able to do it within RealityKit. Thanks in advance.

Hello everyone! how to put a custom shader in depth only on the floor? I'm trying to use the depth of the scene to put the shader exclusively on the floor, but apparently I'm doing something wrong Links https://www.dropbox.com/s/4ghun92frlcg7hz/IMG_9960.PNG?dl=0 https://www.dropbox.com/home?preview=-2362988581602429186.MP4 PostProcess.metal float linearizeDepth(float sampleDepth, float4x4 viewMatrix) {     constexpr float kDepthEpsilon = 1e-5f;     float d = max(kDepthEpsilon, sampleDepth);     d = abs(-viewMatrix[3][2] / d);     return d; } constexpr sampler textureSampler(address::clamp_to_edge, filter::bicubic); float getDepth(float2 coords, constant InputArgs *args, texture2d<float, access::sample> inDepth,depth2d<float, access::sample> arDepth) {     float2 arDepthCoords = args->orientationTransform * coords + args->orientationOffset;     float realDepth = arDepth.sample(textureSampler, arDepthCoords);     float virtualDepth = linearizeDepth(inDepth.sample(textureSampler, coords)[0], args->viewMatrix);     bool realFragment = (virtualDepth <= FLT_EPSILON);     if (realFragment) { virtualDepth = realDepth; }     return min(virtualDepth, realDepth); } float3 getDirection(float2 screenCoord, constant InputArgs *args) {     float3 top = mix(args->topLeft.xyz, args->topRight.xyz, screenCoord.x);     float3 bottom = mix(args->bottomLeft.xyz, args->bottomRight.xyz, screenCoord.x);     return normalize(mix(bottom, top, screenCoord.y)); } float3 worldCoordsForDepth(float depth, float2 screenCords, constant InputArgs *args) {     float3 centerDirection = getDirection(float2(0.5, 0.5), args);     float3 direction = getDirection(screenCords, args);     float depth2 = depth / dot(direction, centerDirection);     return direction * depth2 + args->viewTranslation.xyz; } [[kernel]] void postProcess(uint2 gid [[thread_position_in_grid]],                  texture2d<half, access::read> inputTexture [[texture(0)]],                  texture2d<float, access::sample> inDepth [[texture(1)]],                  texture2d<half, access::write> outputTexture [[texture(2)]],                  depth2d<float, access::sample> arDepth [[texture(3)]],                  constant InputArgs *args [[buffer(0)]]) {  float2 screenCoords = float2(float(gid[0]) / float(outputTexture.get_width()),                                  float(gid[1]) / float(outputTexture.get_height()));     float rawDepth = getDepth(screenCoords, args, inDepth, arDepth);     float3 worldCoords = worldCoordsForDepth(rawDepth, screenCoords, args);     float depth = rawDepth;     depth = 1 - pow(1 / (pow(depth, args->intensity) + 1), args->falloff);     depth = clamp(depth, 0.0, 1.0);     half4 nearColor = inputTexture.read(gid);     float blend = pow(1 - depth, args->exponent);     half4 color = half4(0.0);  float2 frag = worldCoords.xz;         frag *= 1.0 - 0.2 * cos (frag) * sin (3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.0)).x);         frag *= 5.0;         float random = rand (floor(frag));         float2 black = smoothstep(1.0, 0.8, cos(frag * 3.14159 * 2.0));         float3 finalColor = hsv2rgb (float3 (random, 1.0, 1.0));         finalColor *= black.x * black.y * smoothstep (1.0, 0.0, length (fract (frag) - 0.5));         finalColor *= 0.5 + 0.5 * cos (random + random * args->time + args->time + 3.14159 * 0.5 * inDepth.sample(textureSampler, float2(0.7)).x); color = blend * nearColor + (1.0 - blend) * half4(half3(finalColor), 1.0); } I really hope for help of understanding this one

There is a write function documented in the CoreImage Metal shader reference here: https://developer.apple.com/metal/MetalCIKLReference6.pdf But I'm not sure how to use it. I assumed one would be able to use it on the destination parameter i.e. dest.write(...) but I get the error, "no member named 'write' in 'coreimage::destination'" How do I use this function?

I've created a custom BoxBlur kernel that produces identical results to Apple's built-in box blur (CIBoxBlur) kernel but my custom kernel is orders of magnitude slower. So naturally I am wondering what I'm doing wrong to get such poor performance. Below is my custom kernel in the Metal shading language. Can you spot why it's so slow? The built-in filter performs well so I can only assume it's something I'm doing wrong. #include <CoreImage/CoreImage.h> #import <simd/simd.h> extern "C" { namespace coreimage { float4 customBoxBlurFilterKernel(sampler src) { float2 crd = src.coord(); int edge = 100; int minx = crd.x - edge; int maxx = crd.x + edge; int miny = crd.y - edge; int maxy = crd.y + edge; float4 sums = float4(0,0,0,0); float cnt = 0; // compute average of surrounding rgb values for(int row=miny; row < maxy; row++) { for(int col=minx; col < maxx; col++) { float4 samp = src.sample(float2(col, row)); sums[0] += samp[0]; sums[1] += samp[1]; sums[2] += samp[2]; cnt += 1.; } } return float4(sums[0]/cnt, sums[1]/cnt, sums[2]/cnt, 1); } } }

The documentation says the pixel format of the environmentTexture in an AREnvironmentProbeAnchor is bgra8Unorm_srgb. https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor/2977511-environmenttexture However, when I inspect the pixelFormat property of the MTLTexture it says it's rgba16Float. I'm trying to read the texture out as a PNG, and because it's a 16-bit float image, I'm assuming its color space is CGColorSpace.displayP3, but I'm not 100% sure. The texture looks darker than what I expected. Could it be that the color space is sRGB, but it's 16-bit because it's actually an HDR texture stored as linear RGB? (Tested on iPhone 12, iOS 15)

I’m trying to acquire a point cloud and then capture an image (the capture would stop the point cloud acquisition session). The point cloud acquisition seems easy. This is a well discussed topic: https://developer.apple.com/forums/thread/658109 Furthermore, it would seem possible to store camera pose information from the point cloud with the photo so that data could be used to find a pose. Later, I’d align the 3D point cloud to that 2d image (preferably not using depth images, interpolation, or NNs). I am surprised, but seems really hard to find out the feasibility of such an approach.

I am working on rendering multiple objects in Metalkit. While doing it I can see objects are flickering if they are overlaying each-other. I'm passing the currentRenderPassDescriptor and current drawable in draw method which is calling every time. Please refer screenshot. Please help me out to render objects without flickering.

For running my app on devices with P3 screens, to take advantage of the P3 colors, which MTLPixelFormat is the proper one to use for my MTKView? My app is for iOS, but it’s nice to be able to debug on the simulator or as a Catalyst app. Right now, I’ve been using MTLPixelFormatBGR10_XR_sRGB and MTLPixelFormatBGRA10_XR_sRGB per the recommendation in WWDC 2016 session 605 for iOS support of P3 color. This appears to work fine on iOS devices, but on my Apple silicon Mac, when running as a Catalyst app, the colors sometimes look lighter than they should, but it doesn’t always happen. When I run in the simulator on this Mac, my app crashes. When I was using an Intel Mac (with an sRGB screen), I was having my app render in sRGB-only with MTLPixelFormat.bgra8Unorm. MTLPixelFormat.bgra8Unorm also works fine on my Apple silicon Mac, but of course, I’m not getting the P3 color space.  For Apple Silicon Macs, as the GPU hardware is more like an iOS device than like an Intel Mac, should the same pixel formats work on an Apple silicon Mac as on iOS devices? Or are the pixel formats that work with Apple Silicon Macs the same as those on Intel Macs?

Hi, i have a SceneKit/ARKit (with my own metal shader) app and I need to change the main texture format from .bgra8unorm_srgb to .bgra8unorm. Since it is srgb, anything I draw is again converted to srgb and shown too bright. Is there a way to change it/how can it be changed? I didn't find anything useful in docs or examples. Thx

Friendly greetings ! I'm on a MacBook Air M1, using Xcode 13.1, Swift. The full source code is below, as a MaOS Console Application import Foundation import Metal import MetalKit import simd guard let device : MTLDevice = MTLCreateSystemDefaultDevice() else { fatalError("could not create metal default device") } let queue = device.makeCommandQueue() print("Hello, World!") The full output, running in debug from XCode is : 2021-10-29 06:00:16.567089+0200 FraCompute[71702:11572598] Metal API Validation Enabled 2021-10-29 06:00:16.621413+0200 FraCompute[71702:11572598] flock failed to lock list file (/var/folders/s7/7m1rt8kx3jq7c02mwclmnvf80000gn/C//com.apple.metal/16777235_322/functions.list): errno = 35 2021-10-29 06:00:16.628053+0200 FraCompute[71702:11572598] +[MTLIOAccelDevice registerDevices]: Zero Metal services found Hello, World! Program ended with exit code: 0 Running the sample "MacOS Game" default code (the one with the rotating cube) also give this "Zero Metal services found". What's a "metal service" anyway ? Googling for it obviously give me unrelated smithing service offer =^_^= It's not even "Metal Service not found" but, "I found Zero Metal Services", whatever that's supposed to mean ? is it "I didn't find any metal service" or is there a thing called "Zero Metal" ? I can query the metal device name and it get me "Apple M1". No problem, everything seems to work fine (so far). But what's the meaning of this message please ?

Hi all, though I have the YCbCr texture of the camera feed, I still lack the right matrix to convert it to what ARKit is showing as background texture. I tried it with this matrix (found here): constant const float4x4 ycbcrToRGBTransform = float4x4(     float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),     float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),     float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),     float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f) ); But this one is too bright. Are there docs about this or is the background_video_frag source available? (Xcode tells me it's not/needs to be loaded, but I don't know from where) I tried to find out from the buffers used in background_video_frag, but without the source it's hard to tell what's what.

Hi, I have been getting the following crash in my app which has a Custom Metal Render Engine CoreFoundation                   -[__NSSetM clumpingFactor] + 264 libobjc.A.dylib                  __objc_empty_cache + 888 CoreAutoLayout                   DA979160-E330-3C35-BF6F-D3248DCC3246 + 67536 CoreAutoLayout                   DA979160-E330-3C35-BF6F-D3248DCC3246 + 68272 UIKitCore                        __OBJC_$_INSTANCE_METHODS__UIDatePickerCalendarTimeLabel + 600 UIKitCore                        __OBJC_$_INSTANCE_METHODS__UINavigationBarStarkVisualStyle + 100 UIKitCore                        ___79+[UISwitchModernVisualElement _modernThumbImageWithColor:mask:traitCollection:]_block_invoke_2 + 204 UIKitCore                        -[UISwitchModernVisualElement _switchTrackPositionAnimationWithFromValue:toValue:on:] + 388 UIKitCore                        -[UISwitchModernVisualElement _effectiveGradientImage] + 128 UIKitCore                        __OBJC_$_INSTANCE_METHODS__UISearchBarVisualProviderLegacy + 1924 QuartzCore                       CA::Layer::add_animation(CAAnimation*, __CFString const*) + 72 QuartzCore                       CA::Layer::remove_sublayer(CA::Transaction*, CALayer*) + 272 QuartzCore                       CA::OGL::Context::draw_elements(CA::OGL::PrimitiveMode, unsigned int, unsigned short const*, CA::OGL::Vertex const*, unsigned int, unsigned int, CA::OGL::ClipPlane const*) + 60 QuartzCore                       CAML::cgcolor_end(CAML::Context*, CAML::State*, char*, unsigned long) + 1252 QuartzCore                       native_window_swap(_EAGLNativeWindowObject*, unsigned int, double) + 712 QuartzCore                       -[CAStateControllerAnimation initWithLayer:key:] + 52 CoreFoundation                   ___CFSocketSetSocketReadBufferAttrs + 444 CoreFoundation                   __CFNonObjCEqual + 8 CoreFoundation                   __CFRelease + 952 Foundation                       4E7D1FF6-6B64-3833-9E60-CC662AFE2647 + 36236 danmu                            -[DMEngineBase runMetalThread] (in DMEngineBase.mm:148) Foundation                       4E7D1FF6-6B64-3833-9E60-CC662AFE2647 + 1549068 libsystem_pthread.dylib          _pthread_rwlock_unlock$VARIANT$armv81 + 160 libsystem_pthread.dylib          __pthread_create + 1196 this crash seems caused by the Metal Thread Rendering which triggered a CoreAnimation drawing and finally crashed at CoreAutoLayout internal method : Crashed View : appEnterForeground Exception Name : NSInternalInconsistencyException Exception Reason : Modifications to the layout engine must not be performed from a background thread after it has been accessed from the main thread. In the latest Version I have been trying solved this crash by add some protected like this : - (void)runMetalThread {   NSRunLoop *runLoop = [NSRunLoop currentRunLoop];   [_displayLink addToRunLoop:runLoop forMode:DMMetalRunLoopModelFrame];   BOOL continueRunLoop = YES;   while (continueRunLoop)   {     @autoreleasepool     {       [runLoop runMode:DMMetalRunLoopModelFrame beforeDate:[NSDate distantFuture]];     }     continueRunLoop = _continueRunLoop;   } } - (void)onBulletDraw:(CADisplayLink*)displayLink {   self.renderer.stoped = !_isActive || _stoped;   self.renderer.paused = _isPaused;   [self.renderer onDanmuDraw:displayLink]; } #pragma mark - Notification - (void)willResignActive {   self.isActive = NO;   MTLog(@"metal# engine %@ resign active", self); } - (void)didBecomeActive {   //Protected by delay active the metal engine after app state didBecomeActive   dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.7 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{     if (UIApplication.sharedApplication.applicationState == UIApplicationStateActive) {       self.isActive = YES;       MTLog(@"metal# engine %@ become active", self);     }else{       MTLog(@"metal# engine still inactive after 0.7s");     }   }); } this protected has make sure to STOPed the Metal Render after the app resign active, and seems has some effect which reduced 84% crashes on my released app . but it did not completely solve this problem still has 16% crashes in some unknown scene . I have been checked all my Metal Rendering thread code , I can be guaranteed that no method will trigger CoreAnimation drawing in this metal thread. So is there has an Perfective Solution of this bug ?

I've built a lightweight app that displays some advanced animated graphics UI using the Metal Kit Unfortunately, the app launches too slow. The app's icon in the Dock bounces 3-5 times, the icon's bounce animation is jerky, and also the whole Mac becomes temporarily sluggish and may become much less responsive for a few seconds. So it's not merely a matter of my app's launch-time optimization. Also I don't have this problem with other apps that don't use Metal. I narrowed down the problem and figured that the problem disappears if I comment out the code starting from -[CAMetalLayer setDevice:] in my -[NSApplicationDelegate applicationWillFinishLaunching:]. I also noticed in the Activity Monitor that when I launch my app, the kernel_task instantly becomes hugely active, taking up 100 or 200% of CPU, and quickly goes back to 3-4% after the launch is complete. This apparently causes the whole Mac to become sluggish temporarily. I have a theory that for some reason, macOS maxes out the kernel_task when my app calls the GPU for initialization. I know that the kernel_task is used to throttle CPU to avoid overheating, but my Macbook has normal temperature and no other CPU-intense tasks are running. I am running MacBook Pro (16-inch, 2019) 2.6 GHz 6-Core Intel Core i7 16 GB 2667 MHz DDR4 AMD Radeon Pro 5300M 4 GB Intel UHD Graphics 630 1536 MB Can someone please advise about the possible causes of this problem and how to deal with it? My app is extremely lightweight and I really want it to launch instantly.

I notice that when I open the Photos app on my iPhone 12 Pro, viewing Photos or Videos shot in HDR makes them brighter than the overall display brightness level. On macOS, there are APIs like EDRMetadata on CAMetalLayer and maximumExtendedDynamicRangeColorComponentValue on NSScreen. I did see CAMetalLayer.wantsExtendedDynamicRangeContent, but I'm not sure if this does what I'm looking for. The "Using Color Spaces to Display HDR Content" - https://developer.apple.com/documentation/metal/drawable_objects/displaying_hdr_content_in_a_metal_layer/using_color_spaces_to_display_hdr_content?language=objc documentation page describes setting the .colorspace on the CAMetalLayer for BT2020_PQ content, but it's not clear if this is referring to macOS or iOS. Is that the right way to get colors to be "brighter" than 1.0 on "XDR" mobile displays?