Hello everyone! After some time to think about I proceed with graphics api, I figured opengl will be my first since I'm completely new to graphics programming. As in my last post you may find, I was speaking on moltenvk and might just use metal instead, along with the demos I found using metal. So for now, and I know this is said MANY TIMES, apple deprecated opengl but wish to use it because I'm new to graphics programming and want to develop an app(a rendering engine really) for the iPhone 14 Pro Max and macOS Ventura 13.2(I think this is the latest). So what do you guys think? Can I still use opengl es on the 14 max, along with opengl 4+ on latest macOS even though is deprecated?

Hi, I am generating a Metal library that I build using the command line tools on macOS for iphoneos, following the instructions here. Then I serialise this to a binary blob that I load at runtime, which seems to work ok as everything renders as expected. When I am doing a frame capture and open up a shader function it tries to load the symbols and fails. I tried pointing it to the directory (and the file) containing the symbols file, but it never resolves those. In the bottom half of the Import External Sources dialogue there is one entry in the Library | Debug Info section: The library name is Library 0x21816b5dc0 and below Debug Info it says Invalid UUID. The validation layer doesn't flag any invalid behaviour so I am a bit lost and not sure what to try next?

Hello, In one of my apps, I'm trying to modify the pixel buffer from a ProRAW capture to then write the modified DNG. This is what I try to do: After capturing a ProRAW photo, I work in the delegate function func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { ... } In here I can access the photo.pixelBuffer and get its base address: guard let buffer = photo.pixelBuffer else { return } CVPixelBufferLockBaseAddress(buffer, []) let pixelFormat = CVPixelBufferGetPixelFormatType(buffer) // I check that the pixel format corresponds with ProRAW . This is successful, the code enters the if block if (pixelFormat == kCVPixelFormatType_64RGBALE) { guard let pointer = CVPixelBufferGetBaseAddress(buffer) else { return } // We have 16bits per component, 4 components let count = CVPixelBufferGetWidth(buffer) * CVPixelBufferGetHeight(buffer) * 4 let mutable = pointer.bindMemory(to: UInt16.self, capacity: count) // As a test, I want to replace all pixels with 65000 to get a white image let finalBufferArray : [Float] = Array.init(repeating: 65000, count: count) vDSP_vfixu16(finalBufferArray, 1, mutable, 1, vDSP_Length(finalBufferArray.count)) // I create an vImage Pixel buffer. Note that I'm referencing the photo.pixelBuffer to be sure that I modified the underlying pixelBuffer of the AVCapturePhoto object let imageBuffer = vImage.PixelBuffer<vImage.Interleaved16Ux4>(referencing: photo.pixelBuffer!, planeIndex: 0) // Inspect the CGImage let cgImageFormat = vImage_CGImageFormat(bitsPerComponent: 16, bitsPerPixel: 64, colorSpace: CGColorSpace(name: CGColorSpace.displayP3)!, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue | CGBitmapInfo.byteOrder16Little.rawValue))! let cgImage = imageBuffer.makeCGImage(cgImageFormat: cgImageFormat)! // I send the CGImage to the main view controller. This is successful, I can see a white image when rendering the CGImage into a UIImage. This lets me think that I successfully modified the photo.pixelBuffer firingFrameDelegate?.didSendCGImage(image: cgImage) } // Now I try to write data. Unfortunately, this does not work. The photo.fileDataRepresentation() writes the data corresponding to the original, unmodified pixelBuffer `if let photoData = photo.fileDataRepresentation() { // Sending the data to the view controller and rendering it in the UIImage displays the original photo, not the modified pixelBuffer firingFrameDelegate?.didSendData(data: photoData) thisPhotoData = photoData }` CVPixelBufferUnlockBaseAddress(buffer, []) The same happens if I try to write the data to disk. The DNG file displays the original photo and not the data corresponding to the modified photo.pixelBuffer. Do you know why this code should not work? Do you have any ideas on how I can modify the ProRAW pixel buffer so that I can write the modified buffer into a DNG file? My goal is to write a modified file, so, I'm not sure I can use CoreImage of vImage to output a ProRAW file.

I'm trying to debug my metal shaders in Xcode 14.2. However clicking "Capture metal GPU" while debugging recently started showing the following error: Capturing MTLPipelineLibrary is not supported. Unsupported method: -[MTLDevice newPipelineLibraryWithFilePath:error:] To enable capturing, disable calls to unsupported APIs and relaunch your application. I can't find any info about MTLPipelineLibrary or how to disable it. I've also confirmed that Metal GPU Frame Capture is enabled in my build What's causing this issue and how can I work around it so I can debug my shaders again?

I think Monterey 12.6.4 produces deformed sphere and cylinder meshes. Previous Monterey and last and latest Ventura do not have the issue.

After building jaxlib as per the instructions and installing jax-metal, upon testing upon an existing model which works fine using CPU (and GPU on linux), I get the following error. jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: /Users/adam/Developer/Pycharm Projects/gpy_flow_test/sparse_gps.py:66:0: error: failed to legalize operation 'mhlo.cholesky' /Users/adam/Developer/Pycharm Projects/gpy_flow_test/sparse_gps.py:66:0: note: called from /Users/adam/Developer/Pycharm Projects/gpy_flow_test/sparse_gps.py:66:0: note: see current operation: %406 = "mhlo.cholesky"(%405) {lower = true} : (tensor<50x50xf32>) -> tensor<50x50xf32> A have tried to reproduce this with the following minimal example, but this works fine. from jax import jit import jax.numpy as jnp import jax.random as jnr import jax.scipy as jsp key = jnr.PRNGKey(0) A = jnr.normal(key, (100,100)) def calc_cholesky_decomp(test_matrix): psd_test_matrix = test_matrix @ test_matrix.T col_decomp = jsp.linalg.cholesky(psd_test_matrix, lower=True) return col_decomp calc_cholesky_decomp(A) jitted_calc_cholesky_decomp = jit(calc_cholesky_decomp) jitted_calc_cholesky_decomp(A) I am unable to attach the full error message has it exceeds all the restricts placed on uploads attached to a post. I am more than happy to try a more complex model if you have any suggestions.

There is a project tutorial for visionOS Metal rendering in immersive mode here (https://developer.apple.com/documentation/compositorservices/drawing_fully_immersive_content_using_metal?language=objc), but there is no downloadable sample project. Would Apple please provide sample code? The set-up is non-trivial.

Is MTKView intentionally unavailable on visionOS or is this an issue with the current beta?

hello, when I do Metal drawing into an MTKView in full screen, there's an issue with frame scheduling, it seems. There is visible stutter, and the Metal HUD shows the frame rate jittering about. happens in Ventura and Sonoma b2 on my Macbook Pro. here's a really minimal example. Not even actively drawing anything, just presenting the drawable. #import "ViewController.h" #import <Metal/Metal.h> #import <MetalKit/MetalKit.h> @interface ViewController() <MTKViewDelegate> @property (nonatomic, weak) IBOutlet MTKView *mtkView; @property (nonatomic) id<MTLDevice> device; @property (nonatomic) id<MTLCommandQueue> commandQueue; @end @implementation ViewController - (void)viewDidLoad { [super viewDidLoad]; _device = MTLCreateSystemDefaultDevice(); _mtkView.device = _device; _mtkView.delegate = self; _commandQueue = [_device newCommandQueue]; } - (void)drawInMTKView:(MTKView *)view { MTLRenderPassDescriptor *viewRPD = view.currentRenderPassDescriptor; if(viewRPD) { id<MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer]; [commandBuffer presentDrawable:view.currentDrawable]; [commandBuffer commit]; } } - (void)mtkView:(MTKView *)view drawableSizeWillChange:(CGSize)size { NSLog(@"%@", NSStringFromSize(size)); } Looks like there's some collision between display and render timer, or something. what gives? I would like to be able to render stutter free on this very nice machine? how would I go about that?

I am processing CVPixelBuffers received from camera using both Metal and CoreImage, and comparing the performance. The only processing that is done is taking a source pixel buffer and applying crop & affine transforms, and saving the result to another pixel buffer. What I do notice is CPU usage is as high a 50% when using CoreImage and only 20% when using Metal. The profiler shows most of the time spent is in CIContext render: let cropRect = AVMakeRect(aspectRatio: CGSize(width: dstWidth, height: dstHeight), insideRect: srcImage.extent) var dstImage = srcImage.cropped(to: cropRect) let translationTransform = CGAffineTransform(translationX: -cropRect.minX, y: -cropRect.minY) var transform = CGAffineTransform.identity transform = transform.concatenating(CGAffineTransform(translationX: -(dstImage.extent.origin.x + dstImage.extent.width/2), y: -(dstImage.extent.origin.y + dstImage.extent.height/2))) transform = transform.concatenating(translationTransform) transform = transform.concatenating(CGAffineTransform(translationX: (dstImage.extent.origin.x + dstImage.extent.width/2), y: (dstImage.extent.origin.y + dstImage.extent.height/2))) dstImage = dstImage.transformed(by: translationTransform) let scale = max(dstWidth/(dstImage.extent.width), CGFloat(dstHeight/dstImage.extent.height)) let scalingTransform = CGAffineTransform(scaleX: scale, y: scale) transform = CGAffineTransform.identity transform = transform.concatenating(scalingTransform) dstImage = dstImage.transformed(by: transform) if flipVertical { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: 1, y: -1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: 0, y: dstImage.extent.size.height)) } if flipHorizontal { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: -1, y: 1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: dstImage.extent.size.width, y: 0)) } var dstBounds = CGRect.zero dstBounds.size = dstImage.extent.size _ciContext.render(dstImage, to: dstPixelBuffer!, bounds: dstImage.extent, colorSpace: srcImage.colorSpace ) Here is how CIContext was created: _ciContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!, options: [CIContextOption.cacheIntermediates: false]) I want to know if I am doing anything wrong and what could be done to lower CPU usage in CoreImage?

So I have been working on using the new Metal Shader Converter to create a graphics abstraction between D3D12 and Metal. One thing I cannot wrap my head around is how someone would do bindless buffers in Metal. Take this for example... the metal shader converter easily converts this code into Metal `ByteAdressBuffer bindless_buffers[] : register(space1); v2f vertMain(vertIn in){ Mesh m = bindless_buffers[1].Load(0); v2f out; out.pos = in.pos * m.pos; return out;` And using the new Shader Converter one can easily create a DescriptorTable from a root signature that holds this unbounded array of ByteAdressBuffers. But when you try to fill an argument buffer using the DescriptorTableEntry struct, it looks like you can only place one resource at a time and cannot even access the array in the descriptor. For textures this is okay because you can supply a MTLTexture that holds other textures. But you can't have a MTLBuffer hold different kinds of buffers. Is it possible to do this ByteAdressBuffer style of full bindless in Metal? The shader converter allows it but I don't know how to get the API to supply the shaders with the correct data... Any Help would be GREATLY appreciated and all the work I do will be posted online for others to learn about using Metal :)

Hello! I have to use a specific pattern with pointers for a shader and I am not sure what's wrong. The shader renders with artefacts. Seems to be something messed up with the pointers and the UVs. Here is a simplified version: float3 outColor; float2 uv; }; device Context *ContextInit(float3 color, float2 uv) { device Context *context = nullptr; context->outColor = color; context->uv = uv; return context; } void drawSomething(device Context &context) { float d = length(context.uv); context.outColor *= d; } void manupulateUV(device Context &context, float2 uv) { uv +=0.5; float d = length(sin(uv)); context.outColor -=d; } fragment float4 pointer(VertexOut input[[stage_in]]) { float2 uv = input.textureCoordinate; device Context *context = ContextInit(float3(1, 0, 0), uv); drawSomething(*context); return float4(context->outColor.x, context->outColor.y, 0, 1); }

It appears that some of the jax core functions (in pjit, mlir) are not supported. Is this something to be supported in the future? For example, when I tested a diffrax example, from diffrax import diffeqsolve, ODETerm, Dopri5 import jax.numpy as jnp def f(t, y, args): return -y term = ODETerm(f) solver = Dopri5() y0 = jnp.array([2., 3.]) solution = diffeqsolve(term, solver, t0=0, t1=1, dt0=0.1, y0=y0) It generates an error saying EmitPythonCallback is not supported in metal. File ~/anaconda3/envs/jax-metal-0410/lib/python3.10/site-packages/jax/_src/interpreters/mlir.py:1787 in emit_python_callback raise ValueError( ValueError: `EmitPythonCallback` not supported on METAL backend. I uderstand that, currently, no M1 or M2 chips have multiple devices or can be arranged like that. Therefore, it may not be necessary to fully implement p*** functions (pmap, pjit, etc). But some powerful libraries use them. So, it would be great if at least some workaround for core functions are implemented. Or is there any easy fix for this?

I am new to Metal. I need to port OpenCL compute shader programs to Metal compute shaders. I am having trouble in finding sample codes in Metal and Swift, Objective-C. I can see examples with GPU buffer objects only. As in the following OpenCL shader function, I need to pass uniform constant float and integer values along with GPU buffer pointers. I only use compute shaders. __kernel void testfunction ( float ratio1, int opr1, int opr2, __global float *INPUT1, __global float *INPUT2, __global float *OUTPUT } { int peIndex = get_global_id(0); // main compute block } How can I code these in Metal? And how can I set/pass these parameter values in Swift and Objective-c main programs?

I'm trying to compare two allocation schemes in my Metal renderer: allocate separate MTLBuffers out of an MTLHeap allocate one full-size MTLBuffer from an MTLHeap, then doing my own suballocating and tracking the offsets The first is straightforward and I get a nice overview of everything in the Xcode Metal debugger. For the second, I use addDebugMarker:range: to label each of my custom buffers. I have looked everywhere and can't see where my debug labels are supposed to appear in the debugger. The memory overview only shows the one MTLBuffer that spans the entire MTLHeap. My renderpass works as expected, but the command list and resource views only reference the single MTLBuffer as opposed to the tagged ranges. What am I missing?

Hi, I'm following https://developer.apple.com/metal/jax/ to install jax on my Mac. The installation is successful. However, running the give example gives $ python -c 'import jax; jax.numpy.arange(10)' 2023-07-27 20:26:08.492162: W pjrt_plugin/src/mps_client.cc:535] WARNING: JAX Apple GPU support is experimental and not all JAX functionality is correctly supported! Metal device set to: Apple M2 Pro systemMemory: 16.00 GB maxCacheSize: 5.33 GB loc("-":2:3): error: custom op 'func.func' is unknown fish: Job 1, 'python3 $argv' terminated by signal SIGSEGV (Address boundary error)

I am working on creating a "Volume" application in RealityKit for visionOS. I want to create a texture on the CPU that I can hook into a Material and modify. When I go to create the texture I get this error: Linear textures from shared buffers is not supported on this device Here is the code: guard let device = MTLCreateSystemDefaultDevice() else { fatalError("unable to get metal device") } let textureDescriptor = MTLTextureDescriptor.textureBufferDescriptor(with: .r32Float, width: 64, resourceOptions: .storageModeShared, usage: .shaderRead) let buffer = device.makeBuffer(length: 64 * 4) return buffer?.makeTexture(descriptor: textureDescriptor, offset: 0, bytesPerRow: 64 * 4)

I have a following MTLBuffer created. How can I send INPUTVALUE to the memINPUT buffer? I need to send repeatedly in Objective-C. // header file @property id<MTLBuffer> memINPUT; // main file int length = 1000; ... memINPUT = [_device newBufferWithLength:(sizeof(float)*length) options:0]; ... float INPUTVALUE[length]; for (int i=0; i < length; i++) { INPUTVALUE[i] = (float)i; } // How to send to INPUTVALUE to memINPUT? ... The following is Swift version. I am looking for Objective-c version. memINPUT.contents().copyMemory(from: INPUTVALUE, byteCount: length * MemoryLayout<Float>.stride);

I'm trying to save metal textures in a lossless compressed format. I've tried png and tiff, but I run into the same problem: the pixel data changes after save and load when we have transparency. Here is the code I use to save a Tiff: import ImageIO import UIKit import Metal import MobileCoreServices extension MTLTexture { func saveAsLosslessTIFF(url: URL) throws { guard let context = CIContext() else { return } guard let colorSpace = CGColorSpace(name: CGColorSpace.linearSRGB) else { return } guard let ciImage = CIImage(mtlTexture: self, options: [.colorSpace : colorSpace]) else { return } guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { return } // create a dictionary with TIFF compression options let tiffCompression_LZW = 5 let options: [String: Any] = [ kCGImagePropertyTIFFCompression as String: tiffCompression_LZW, kCGImagePropertyDepth as String: depth, kCGImagePropertyPixelWidth as String: width, kCGImagePropertyPixelHeight as String: height, ] let fileDestination = CGImageDestinationCreateWithURL(url as CFURL, kUTTypeTIFF, 1, nil) guard let destination = fileDestination else { throw RuntimeError("Unable to create image destination.") } CGImageDestinationAddImage(destination, cgImage, options as CFDictionary) if !CGImageDestinationFinalize(destination) { throw RuntimeError("Unable to save image to destination.") } } } I can then load the texture like this: func loadTexture(url:URL) throws -> MTLTexture { let usage = MTLTextureUsage(rawValue: MTLTextureUsage.renderTarget.rawValue | MTLTextureUsage.shaderRead.rawValue | MTLTextureUsage.shaderWrite.rawValue) return try loader.newTexture(URL:url,options:[MTKTextureLoader.Option.textureUsage:usage.rawValue,MTKTextureLoader.Option.origin:MTKTextureLoader.Origin.flippedVertically.rawValue]) } After saving and then loading the texture again, I want to get back the exact same texture. And I do, if there is no transparency. Transparent pixels, however, are transformed in a way that I don't understand. Here is an example pixel: [120, 145, 195, 170] -> [144, 174, 234, 170] My first guess would be that something is trying to undo a pre-multiplied alpha that never happened. But the numbers don't seem to work out. For example, if that were the case I'd expect 120 to go to (120 * 255) / 170 = 180 , not 144. Any idea what I am doing wrong?

As title desc. Before iOS 16.6 is smooth. But iOS 16.6 is lagging. Seems iOS 17 beta4 is ok.. Maybe Apple should post a hotfix for iOS 16.6