Render advanced 3D graphics and perform data-parallel computations using graphics processors using Metal.

Posts under Metal tag

199 Posts
Sort by:

Post

Replies

Boosts

Views

Activity

HDR video metadata
On an iOS 18 phone, I use AVCaptureSession to capture HDR with x420 format. The output CMSampleBuffer is HLG colorspace, the propagated attachments contain kCVImageBufferAmbientViewingEnvironmentKey and kCVImageBufferSceneIlluminationKey. Now I use CAMetalLayer to render the CVPixelBuffer to the screen, but the brightness is brighter than AVSampleBufferDisplayLayer. Here is my code. - (void)_updateColorSpaceIfNeed:(CVPixelBufferRef)pixelBuffer { CAMetalLayer *layer = (CAMetalLayer *)_mtkView.layer; if (![layer isKindOfClass:CAMetalLayer.class]) return; layer.wantsExtendedDynamicRangeContent = YES; CFDataRef ambientViewingEnvironment = (CFDataRef)CVBufferCopyAttachment(pixelBuffer, kCVImageBufferAmbientViewingEnvironmentKey, NULL); NSData *data = (__bridge NSData *)ambientViewingEnvironment; if (ambientViewingEnvironment) CFRelease(ambientViewingEnvironment); CAEDRMetadata *metadata = [CAEDRMetadata HLGMetadataWithAmbientViewingEnvironment:data]; // CAEDRMetadata *metadata = [CAEDRMetadata HLGMetadata]; layer.EDRMetadata = metadata; layer.pixelFormat = MTLPixelFormatRGBA16Float; CGColorSpaceRef colorspace = CGColorSpaceCreateWithName(kCGColorSpaceITUR_2100_HLG); layer.colorspace = colorspace; if (colorspace) CGColorSpaceRelease(colorspace); } Why does the CAEDRMetadata class have "HLGMetadataWithAmbientViewingEnvironment:" and "HLGMetadata" methods, but does not provide the "HLGMetadataWithAmbientViewingEnvironment:sceneIllumination" method? I want to know how kCVImageBufferAmbientViewingEnvironmentKey and kCVImageBufferSceneIlluminationKey affect tone mapping. Is there any documentation I can refer to?
0
0
74
1d
Metal: Non-uniform thread groups unsupported in Simulator? Is it?
My app is running Compute Shaders that use non-uniform thread groups. When I run the app in the debugger with a simulator target the app crashes on encoder.dispatchThreads and the error message is: Dispatch Threads with Non-Uniform Threadgroup Size is not supported on this device. Previously the log output states that: Metal Shader Validation is unsupported for Simulator. However: When I stop the debugger and just run the app in the simulator without the debugger attached, the app just runs fine and does not crash. The SwiftUI Preview that also triggers the Compute Shader when preparing data also just runs fine without a crash. I can run and debug on a real device no problem - I just don't have all sizes available. Is there anything I need to check in my lldb/simulator configuration? It obviously does work, just the debugger cannot really deal with it? Any input would be nice as this really slows my down as I have to be extremely careful when debugging on the simulator.
1
0
152
1d
Is Metal usable from Swift 6?
Hello ladies and gentlemen, I'm writing a simple renderer on the main actor using Metal and Swift 6. I am at the stage now where I want to create a render pipeline state using asynchronous API: @MainActor class Renderer { let opaqueMeshRPS: MTLRenderPipelineState init(/*...*/) async throws { let descriptor = MTLRenderPipelineDescriptor() // ... opaqueMeshRPS = try await device.makeRenderPipelineState(descriptor: descriptor) } } I get a compilation error if try to use the asynchronous version of the makeRenderPipelineState method: Non-sendable type 'any MTLRenderPipelineState' returned by implicitly asynchronous call to nonisolated function cannot cross actor boundary Which is understandable, since MTLRenderPipelineState is not Sendable. But it looks like no matter where or how I try to access this method, I just can't do it - you have this API, but you can't use it, you can only use the synchronous versions. Am I missing something or is Metal just not usable with Swift 6 right now?
0
0
232
4d
Implementing multi-pass rendering in VisionOS
I’m working on a Vision Pro app using Metal and need to implement multi-pass rendering. Specifically, I want to render intermediate results to a texture, then use that texture in a second pass for post-processing before presenting the final output. What’s the best approach in visionOS? Should I use multiple render passes in a single command buffer or separate command buffers? Any insights on efficiently handling this in RealityKit or Metal? Thanks!
0
0
114
5d
Implementing Scalable Order-Independent Transparency (OIT) in Metal
Hi, Apple’s documentation on Order-Independent Transparency (OIT) describes an approach using image blocks, where an array of size 4 is allocated per fragment to store depth and color in a tile shading compute pass. However, when increasing the scene’s depth complexity by adding more overlapping quads, the OIT implementation fails due to the fixed array size. Is there a way to dynamically allocate storage for fragments based on actual depth complexity encountered during rasterization, rather than using a fixed-size array? Specifically, can an adaptive array of fragments be maintained and sorted by depth, where the size grows as needed instead of being limited to 4 entries? Any insights or alternative approaches would be greatly appreciated. Thank you!
0
0
230
6d
Regarding AR App Submission Built in Xcode - Swift Student Challenge submission
Hello guys, I have a question regarding the submission requirements. My app uses ARKit and requires Metal files for shaders, which are not supported by Swift Playgrounds. Therefore, I developed my app using Xcode. (swift playgrounds returning error for metal file) Since my app relies on a real device for proper functionality, I would like to know if, under these circumstances, the scene build is performed by Xcode. If the build were instead done by Swift Playgrounds, my scene would not function correctly. I'm asking that because of this note Thank you for your time and assistance.
1
0
258
6d
Metal not working in Swift Playgrounds (SSC Scene)
Hi everyone, I'm currently working on a Swift Playgrounds project where I need to incorporate a Metal shader file. However, when I tried to include my shader file (PincushionShader.metal), I encountered the following error: Is it possible to use Metal shader files within Swift Playgrounds, it is really important for my swift student challenge scene? If not, are there any workarounds or recommended approaches for testing Metal shaders in a similar environment? Any guidance or suggestions would be greatly appreciated!
3
0
259
5d
Trouble Loading Precompiled Metal Shader (.metallib) into ShaderLibrary
I am currently finalizing my Swift Student Challenge submission, and Metal shaders are an essential part of my app. However, during submission, I noticed a note explaining: "Note: Xcode app playgrounds are run in Simulator", which is not possible for my app, as it also requires the camera of a physical device to function. So, I am currently transferring my app from Xcode into Swift Playgrounds, which I presume will run on physical devices. However, I noticed that Swift Playgrounds do not yet support Metal shaders directly, so I am now pre-compiling my shaders to load them at runtime instead. Note that all the code below was run either in the terminal or in Xcode. I have already compiled my Metal shaders with: xcrun -sdk iphoneos metal -o Shaders.ir -c Shaders.metal xcrun -sdk iphoneos metallib Shaders.ir -o Shaders.metallib Which seems to have run without any problems. When I run: let shaderPath = Bundle.main.path(forResource: "Shaders", ofType: "metallib") let shaderURL = URL(fileURLWithPath: shaderPath!) let shaderData = try! Data(contentsOf: shaderURL) do { let device = MTLCreateSystemDefaultDevice()! let library = try shaderData.withUnsafeBytes { bytes -> MTLLibrary? in let dispatchData = DispatchData(bytes: bytes) return try device.makeLibrary(data: dispatchData as __DispatchData) } print(library!.functionNames) } catch { print(error.localizedDescription) } My Metal shader functions are printed correctly in the console. However, based on my research, it seems like a MTLLibrary cannot be converted into a SwiftUI ShaderLibrary. That is why I am now looking at these two initializers: ShaderLibrary(url: URL) ShaderLibrary(data: Data) Which state: Creates a new Metal shader library from the contents of url/data, which must be the contents of precompiled Metal library. Functions compiled from the returned library will only be cached as long as the returned library exists., which I believe should work for my use case. However, the problem arises when I run this code: let shaderPath = Bundle.main.path(forResource: "Shaders", ofType: "metallib") let shaderURL = URL(fileURLWithPath: shaderPath!) let library = ShaderLibrary(url: shaderURL) My app consistently seems to crash on the ShaderLibrary initialization, rendering the app unusable. Why does ShaderLibrary(url: shaderURL) cause a crash, even though my .metallib file is valid? Are there additional requirements for loading a ShaderLibrary that I may have missed?
4
0
252
6d
Metal calls hanging/stuck if app is started quickly after login
Our app uses Metal for image processing. We have found that if our app (and its possible intensive image processing) is started quickly after user is logged in, then calls to Metal may be hanging/stuck for a good while. Example: it can take 1-2 minutes for something that usually takes 3-5 seconds! Metal threads are just hanging in a memmove... In Activity Monitor we see a lot of things are happening right after log-in. But why Metal calls are blocking for so long is unknown to us... The workaround is to wait a minute before we start our app and start intensive image processing using Metal. But hard to explain this workaround to end-users... It doesn't happen on all computers but fairly easy to reproduce on some computers. We are using macOS 15.3.1. M1/M3 Max. Any good ideas for how to proceed with this problem and possible reach out to Apple engineers? Thanks! :)
2
0
206
1w
Metal Integration with SwiftUI
Hello! I have asked this question in previous years, but I want to make sure I can be safe as each challenge could be different. Are applicants for the Swift Student Challenge allowed to use the features and technologies involved with Metal/MetalKit? Last year, the answer was yes. I have seen a few people here and there use it with Swift and won. I would like to know if we can use it for the 2025 challenge for this year as well. Thanks! :)
2
0
242
2w
alternative for CustomShader in visionOS
Following the post on https://developer.apple.com/documentation/realitykit/custommaterial it's simple to use shader for materials and get uniforms and params from each vertex. However it's not available for visionOS. Any alternative to use in this case? I want to write shader to fill material by myself. (I have shader experience from web, familiar with fragment shader)
1
0
213
2w
Concurrent conflicting texture writes
Hello! I need to "draw" a set of particles into the texture. It would be trivial in render encoder of course. However, I would like to implement the task in compute kernel. Every particle draw operation is expected to set 5 texels - "center" one and left/right/upper/lower. Particles can and will overlap, so concurrent draws are to be expected. I tried using texture atomics - atomic_store() to be more precise. This worked, albeit pretty slowly - too slow for my purpose. Just to test what would happen, I tried using normal texture write(). I was expecting to see some kind of visual artefacts, but to my surprise, it worked very well (and much faster). My question: is it safe? I understand that calling write() doesn't guarantee any ordering of the operations, so if multiple threads write to the same texel, the final value may come from any of those threads. But suppose all the threads were to write the very same color? Can I assume that the texel in question will have said color after the compute kernel finishes? I am using M2 Pro MacBook, but ideally I would love to get the answer for the all Apple Silicon devices. My texture format is R32Int (so as to be able to use atomics), but I could do with any single-channel format, the purpose of the texture is to be binary mask of sorts. Thanks!
0
0
210
3w
Rendering Order with ModelSortGroup
I have a huge sphere where the camera stays inside the sphere and turn on front face culling on my ShaderGraphMaterial applied on that sphere, so that I can place other 3D stuff inside. However when it comes to attachment, the object occlusion never works as I am expecting. Specifically my attachments are occluded by my sphere (some are not so the behavior is not deterministic. Then I suspect it was the issue of depth testing so I started using ModelSortGroup to reorder the rending sequence. However it doesn't work. As I was searching through the internet, this post's comments shows that ModelSortGroup simply doesn't work on attachments. So I wonder how should I tackle this issue now? To let my attachments appear inside my sphere. OS/Sys: VisionOS 2.3/XCode 16.3
1
0
284
3w
Learn Metal
I am interested in learning the Metal framework for rendering development. However, most of Apple’s official documentation uses Objective-C code. Therefore, I am seeking guidance on whether it is more advantageous for me to focus solely on learning Swift to gain proficiency in Metal.
2
0
511
4w
After updating CAMetalLayer.drawableSize, [CAMetalLayer nextDrawable:] frequently takes ~1s
I have a bare-bones Metal app setup where I attach a CAMetalLayer to a window that inherits from a NSWindow with a custom delegate. Everything else is vanilla. I'm also using metal-cpp and metal shader converter. I'm running into a issue where the application runs fine in the beginning, but once I resize the window, it starts hitching. It turns out that [CAMetalLayer nextDrawable:] frequently (but not always) takes around a full second (plus or minus a few milliseconds) to return once drawableSize has been updated. I've tried setting allowsNextDrawableTimeout to false which doesn't work; it returns a valid drawable after a second instead of nil. Setting displaySyncEnabled to false reduces the likelihood of this happening to around 50% from 90%+ but does not eliminate it. Setting maximumDrawableCount to 2 or 3 does not seem to make a difference. By dumping the resource IDs of the returned textures I've noticed something interesting: Before resizing, the layer seems to shuffle between 2 textures or at least 2 resource IDs, but after resizing it starts to create new textures for each returned drawable. Occasionally it seems to reuse a previous resource ID, but it does not seem to have anything to do with whether the method returns quickly or not. Why does this happen, and how can I fix it? Should I create a new CAMetalLayer when resizing the window instead of updating drawableSize?
3
0
414
3w
Use Metal to conver HDR Pixelbuffer to SDR Pixelbuffer
I see some demo show convert HDR video to SDR Pixelbuffer,such AVAssetReader、 AVVideoComposition 、AVComposition 、AVFoundation. But In some cases,I want to render HDR Pixelbuffer and record video. AVCaptureSession *session = [[AVCaptureSession alloc] init]; session.sessionPreset = AVCaptureSessionPresetHigh; AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; if ([videoDevice isVideoHDRSupported]) { NSError *error = nil; if ([videoDevice lockForConfiguration:&error]) { videoDevice.automaticallyAdjustsVideoHDREnabled = NO; videoDevice.videoHDREnabled = YES; // 开启 HDR [videoDevice unlockForConfiguration]; } else { NSLog(@"Error: %@", error.localizedDescription); } } Real-time processing of HDR data requires processing of video frame data (such as filters), ensuring that the processing chain supports 10-bit color depth and HDR metadata. And use imagesBuffer to object tracking, etc. How to solve this problem?
1
0
230
Jan ’25
CATransaction commit() crashed on background thread [EXC_BREAKPOINT: com.apple.root.****-qos.cooperative]
Problem Description We are developing a app for iOS and iPadOS that involves extensive custom drawing of paths, shapes, texts, etc. To improve drawing and rendering speed, we use CARenderer to generate cached images (CGImage) on a background thread. We adopted this approach based on this StackOverflow post: https://stackoverflow.com/a/75497329/9202699. However, we are experiencing frequent crashes in our production environment that we can hardly reproduce in our development environment. Despite months of debugging and seeking support from DTS and the Apple Feedback platform, we have not been able to fully resolve this issue. Our recent crash reports indicate that the crashes occur when calling CATransaction.commit(). We suspect that CATransaction may not be functioning properly outside the main thread. However, based on feedback from the Apple Feedback platform, we were advised to use CATransaction.begin() and CATransaction.commit() on a background thread. If anyone has any insights, we would greatly appreciate it. Code Sample The line CATransaction.commit() is causing the crash: [EXC_BREAKPOINT: com.apple.root.****-qos.cooperative] private let transactionLock = NSLock() // to ensure one transaction at a time private let device = MTLCreateSystemDefaultDevice()! @inline(never) static func drawOnCGImageWithCARenderer( layerRect: CGRect, itemsToDraw: [ItemsToDraw] ) -> CGImage? { // We have encapsulated everything related to CALayer and its // associated creations and manipulations within CATransaction // as suggested by engineers from Apple Feedback Portal. transactionLock.lock() CATransaction.begin() // Create the root layer. let layer = CALayer() layer.bounds = layerRect layer.masksToBounds = true // Add one sublayer for each item to draw itemsToDraw.forEach { item in // We have thousands or hundred thousands of drawing items to add. // Each drawing item may produce a CALayer, CAShapeLayer or CATextLayer. // This is also why we want to utilise CARenderer to leverage GPU rendering. layer.addSublayer( item.createCALayerOrCATextLayerOrCAShapeLayer() ) } // Create MTLTexture and CARenderer. let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor( pixelFormat: .rgba8Unorm, width: Int(layer.frame.size.width), height: Int(layer.frame.size.height), mipmapped: false ) textureDescriptor.usage = [MTLTextureUsage.shaderRead, .shaderWrite, .renderTarget] let texture = device.makeTexture(descriptor: textureDescriptor)! let renderer = CARenderer(mtlTexture: texture) renderer.bounds = layer.frame renderer.layer = layer.self /* ********************************************************* */ // From our crash report, this is where the crash happens. CATransaction.commit() /* ********************************************************* */ transactionLock.unlock() // Rendering layers onto MTLTexture using CARenderer. renderer.beginFrame(atTime: 0, timeStamp: nil) renderer.render() renderer.endFrame() // Draw MTLTexture onto image. guard let colorSpace = CGColorSpace(name: CGColorSpace.sRGB), let ciImage = CIImage(mtlTexture: texture, options: [.colorSpace: colorSpace]) else { return nil } // Convert CIImage to CGImage. let context = CIContext() return context.createCGImage(ciImage, from: ciImage.extent) }
0
1
247
Jan ’25
Instruments showing incorrect values
Hello, I’m encountering an issue with the Instruments app while running a benchmark on an M2 Ultra Mac Studio. Despite being certain that GPU activities involving memory read and write operations are occurring, all related performance counters consistently return 0. Interestingly, this problem does not occur when using the same code on an M1 MacBook Air, where the counters behave as expected. What could be causing this discrepancy? Any insights or suggestions would be greatly appreciated. Thank you!
0
0
209
Jan ’25