MetalKit in SwiftUI

Hi,

Same question here :

I have successfully displayed a SceneKit View in SwifUI (aka SCNView) using UIViewRepresentable, but failed to do this with MetalKit View (aka MTKView).

Thank you in advance if someone as an idea how to do that.

Here's a condensed implementation without doing that much other than rendering CIFilter on images. I have another MTKView that's actually running a compute pipeline and I banged my head a bit figuring out how to organize the code. Mainly the pattern sorta expect you to instantiate everything and leaving the Coordinator to handle actual delegation. However, for an MTKView, you are stuck either instantiating everything within the Coordinator itself or writing an additional Renderer Coordinator class to instantiate everything within. I didn't feel like going down the separate renderer route, so I just passed MTKView straight into the Coordinator and do the typical setup there.

struct MTKMapView: UIViewRepresentable {
    typealias UIViewType = MTKView
    var mtkView: MTKView
            
    func makeCoordinator() -> Coordinator {
        Coordinator(self, mtkView: mtkView)
    }
    
    func makeUIView(context: UIViewRepresentableContext<MetalMapView>) -> MTKView {
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.backgroundColor = context.environment.colorScheme == .dark ? UIColor.white : UIColor.white
        mtkView.isOpaque = true
        mtkView.enableSetNeedsDisplay = true
        return mtkView
    }
    
    func updateUIView(_ uiView: MTKView, context: UIViewRepresentableContext<MetalMapView>) {
        
    }
    
    class Coordinator : NSObject, MTKViewDelegate {
        var parent: MetalMapView
        var ciContext: CIContext!
        var metalDevice: MTLDevice!

        var metalCommandQueue: MTLCommandQueue!
        var mtlTexture: MTLTexture!
                
        var startTime: Date!
        init(_ parent: MetalMapView, mtkView: MTKView) {
            self.parent = parent
            if let metalDevice = MTLCreateSystemDefaultDevice() {
                mtkView.device = metalDevice
                self.metalDevice = metalDevice
            }
            self.ciContext = CIContext(mtlDevice: metalDevice)
            self.metalCommandQueue = metalDevice.makeCommandQueue()!
            
            super.init()
            mtkView.framebufferOnly = false
            mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
            mtkView.drawableSize = mtkView.frame.size
            mtkView.enableSetNeedsDisplay = true
        }

        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
            
        }
        
        func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            let inputImage = CIImage(mtlTexture: mtlTexture)!
            var size = view.bounds
            size.size = view.drawableSize
            size = AVMakeRect(aspectRatio: inputImage.extent.size, insideRect: size)
            let filteredImage = inputImage.transformed(by: CGAffineTransform(
                scaleX: size.size.width/inputImage.extent.size.width,
                y: size.size.height/inputImage.extent.size.height))
            let x = -size.origin.x
            let y = -size.origin.y
            
            
            self.mtlTexture = drawable.texture
            ciContext.render(filteredImage,
                to: drawable.texture,
                commandBuffer: commandBuffer,
                bounds: CGRect(origin:CGPoint(x:x, y:y), size: view.drawableSize),
                colorSpace: CGColorSpaceCreateDeviceRGB())

            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }
        
        func getUIImage(texture: MTLTexture, context: CIContext) -> UIImage?{
            let kciOptions = [CIImageOption.colorSpace: CGColorSpaceCreateDeviceRGB(),
                              CIContextOption.outputPremultiplied: true,
                              CIContextOption.useSoftwareRenderer: false] as! [CIImageOption : Any]
            
            if let ciImageFromTexture = CIImage(mtlTexture: texture, options: kciOptions) {
                if let cgImage = context.createCGImage(ciImageFromTexture, from: ciImageFromTexture.extent) {
                    let uiImage = UIImage(cgImage: cgImage, scale: 1.0, orientation: .downMirrored)
                    return uiImage
                }else{
                    return nil
                }
            }else{
                return nil
            }
        }
    }
}

Same thing here. It would be great if metal kit was more naturally integrated with SwiftUI, without having to go through Coordinators and the likes

Can you please share the MetalMapView code? Is it some kind of SwiftUI View with a UIHostingController?..

Or is it a typo and you meant MTKMapView instead of MetalMapView ?

Any updates of Metal working with SwiftUI this year?

For completeness, here is a standalone example that also creates its own MTKView. To use this view, call it in ContentView's body like so:

Code Block swift
    var body: some View {
        MetalView()
    }

Here is the [NS/UI]ViewRepresentable class:

Code Block swift
import MetalKit
struct MetalView: NSViewRepresentable {
    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }
    func makeNSView(context: NSViewRepresentableContext<MetalView>) -> MTKView {
        let mtkView = MTKView()
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.enableSetNeedsDisplay = true
        if let metalDevice = MTLCreateSystemDefaultDevice() {
            mtkView.device = metalDevice
        }
        mtkView.framebufferOnly = false
        mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
        mtkView.drawableSize = mtkView.frame.size
        mtkView.enableSetNeedsDisplay = true
        return mtkView
    }
    func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) {
    }
    class Coordinator : NSObject, MTKViewDelegate {
        var parent: MetalView
        var metalDevice: MTLDevice!
        var metalCommandQueue: MTLCommandQueue!
        
        init(_ parent: MetalView) {
            self.parent = parent
            if let metalDevice = MTLCreateSystemDefaultDevice() {
                self.metalDevice = metalDevice
            }
            self.metalCommandQueue = metalDevice.makeCommandQueue()!
            super.init()
        }
        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        }
        func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            let rpd = view.currentRenderPassDescriptor
            rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
            rpd?.colorAttachments[0].loadAction = .clear
            rpd?.colorAttachments[0].storeAction = .store
            let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
            re?.endEncoding()
            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }
    }
}

I recommend watching (https://developer.apple.com/videos/play/wwdc2019/231/) to understand how this works.

The suggested code works great, except I can’t get it to update continuously. It will load perfectly, call draw once, then not call it again unless I’m resizing the window. Is there anyway to call draw continuously?

Nevermind... I just added this in makeNSView and it worked great:

Code Block 
mtkView.isPaused =false

I thought it was false by default.

Thanks for sharing the code. But the code is not working for me, got errors like "Cannot find type NSViewRepresentable in scope".

Here is another resource regarding to UIKit Integration (available in YouTube) into SwiftUI by Stanford.

In response to songyeah's note: "Thanks for sharing the code. But the code is not working for me, got errors like "Cannot find type NSViewRepresentable in scope".

I just tried the posted MetalView struct, and it ran fine with the macOS simulator. However, switching to the iOS simulator, I get the same error that you described. I then changed all of the NSView... commands to instead be UIView... commands, and it now runs with the iOS simulator.

(By the way, this code is very helpful for me!) Thanks.

Hi rthart, thanks for your response. I had exactly the same experience like you, and now it is working. Cheers!

Replying to @crawforb: "The suggested code works great, except I can’t get it to update continuously. It will load perfectly, call draw once, then not call it again unless I’m resizing the window. Is there anyway to call draw continuously?"

I had the same issue. The draw() function in coordinator was only called once. I am wondering if you have solved this problem. Thanks.

oops, solved the problem by adding mtkView.isPaused = false, and commenting other settings.

I am curious how to interface with Metal shaders. The above MetalView code provided by the Graphics & Games engineer works, and works when I implement the Coordinator class in a separate file.

The issue I'm having seems to be when we set the mtkView delegate:

Code Block 
func makeUIView(context: UIViewRepresentableContext<MetalMapView>) -> MTKView {
	mtkView.delegate = context.coordinator
	...
}

because (see MakeCoordinator() https://developer.apple.com/documentation/swiftui/nsviewrepresentable/makecoordinator()-9e4i4 )

SwiftUI calls this method before calling the makeUIView(context:) method.

(of course the same is also true in the case of NSViewRepresentable, makeNSView )

My Coordinator class is just a renderer of sorts, and during initialization requires an MTKView. This is where I am having trouble. I am unsure of how (or where?) to create an MTKView for passing to my Coordinator / MTKViewDelegate. For example, in my Coordinator class, I attempt to obtain the aspect ratio by

Code Block 
var aspect = Float(mtkView.drawableSize.width / mtkView.drawableSize.height)

which returns NaN or some equivalent nonsense. I guess I do not have a properly initialized MTKView or something, I am not entirely sure how or where to do this (disclaimer: I am still fairly new to Metal and Swift programming). Furthermore, I do not know where I should be storing the MTKView -- should it be as @State or @Environment? Should it not be stored or initialized inside the MetalView struct?

I do not have issues when I do the same thing in a non-SwiftUI setup, e.g. in

Code Block 
class ViewController: UIViewController {
    var mtkView: MTKView!
	...
    override func viewDidLoad() {
        super.viewDidLoad()
        guard let mtkViewTemp = self.view as? MTKView else {...} 
        mtkView = mtkViewTemp
		...
}

and as far as I know making the (naive, speaking for myself) switch from UIViewRepresentable to UIViewControllerRepresentable is not a fix. As I stated a few lines above, the issue seems to be with how I am attempting to create, store, and use the MTKView.

Thanks everyone for posting your experiences.

I'm building a photo filter app (hobby project) from scratch for macOS. and got here when researching why my sliders started to become too slow to use and what to do about it.

So I added the example by Graphics and Games Engineer  to my project to replace my "ImageView" and display my CI / CG images directly without converting them to NSImage.

So far I got a working view, which shows a green background, but I am unable to figure out how to display my image on this, what to pass as context to view as it wants an NSViewRepresentableContext and the only thing I have is a CIContext...

I think I want to be able to display filter.outputImage... here's an example of how I make this:

Code Block Swift
func ciExposure (inputImage: CIImage, inputEV: Double) -> CIImage {
    let filter = CIFilter(name: "CIExposureAdjust")!
    filter.setValue(inputImage, forKey: kCIInputImageKey)
    filter.setValue(inputEV, forKey: kCIInputEVKey)
    return filter.outputImage!
}

Can someone give me a hint about how to do this? I don't mind doing research but it starts to feel like I've hit a wall here...

Thanks in advance!

Hi: This thead was quite helpful with my own code, thank you!With the newly announced (WWDC 2023) metal shaders available in SwiftUI, I wonder how the standalone example that creates its own MTKView would be updated with the latest SwiftUI? It is not obvious, at least to me...

Supporting MetalKt in SwiftUI might make things easier, but even currently, rendering a MTKView with SwiftUI is no more or less involved than it is with AppKit. The code ends up structured a little differently, but it's the same code, and more or less the same methods.

I've implemented the same simple pixel shader roughly four different ways now - twice with AppKit, twice with SwiftUI. IMO, the best approach uses SwiftUI, and was derived from the example given above (posted a few years ago, by an anonymous Apple dev). The code turns out very similar either way, but AppKit also uses a storyboard (which makes it feel clunky and magical), and SwiftUI is the future etc, so I went with SwiftUI.

I'll include a minimal example here (for my own future reference, as much as anyone else's). The app's called Console. This is is the entrypoint, named ConsoleApp.swift:

import SwiftUI

@main
struct ConsoleApp: App {

    var body: some Scene { WindowGroup { MetalView() } }
}

#Preview { MetalView() }

The bulk of the Swift code is in a file named MetalView.swift. It initializes everything, including a buffer for passing vertices to the shader. These vertices are used by the vertex function to render a rectangle (formed from two triangles, in a triangle-strip), so the frag shader runs once for each pixel in the framebuffer.

import SwiftUI
import MetalKit

struct MetalView: NSViewRepresentable {

    static private let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
    static private let clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)

    class Coordinator: NSObject, MTKViewDelegate {

        var device: MTLDevice

        private var commandQueue: MTLCommandQueue
        private var pipelineState: MTLRenderPipelineState
        private var vertexBuffer: MTLBuffer

        override init() {

            device = MTLCreateSystemDefaultDevice()!

            let library = device.makeDefaultLibrary()!
            let descriptor = MTLRenderPipelineDescriptor()
            let vertices: [simd_float2] = [[-1, +1],  [+1, +1],  [-1, -1],  [+1, -1]]
            let verticesLength = 4 * MemoryLayout<simd_float2>.stride

            descriptor.label = "Pixel Shader"
            descriptor.vertexFunction = library.makeFunction(name: "init")
            descriptor.fragmentFunction = library.makeFunction(name: "draw")
            descriptor.colorAttachments[0].pixelFormat = MetalView.pixelFormat

            commandQueue = device.makeCommandQueue()!
            pipelineState = try! device.makeRenderPipelineState(descriptor: descriptor)
            vertexBuffer = device.makeBuffer(bytes: vertices, length: verticesLength, options: [])!

            super.init()
        }

        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) { }

        func draw(in view: MTKView) {

            let buffer = commandQueue.makeCommandBuffer()!
            let descriptor = view.currentRenderPassDescriptor!
            let encoder = buffer.makeRenderCommandEncoder(descriptor: descriptor)!

            encoder.setRenderPipelineState(pipelineState)
            encoder.setVertexBuffer(self.vertexBuffer, offset: 0, index: 0)
            encoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
            encoder.endEncoding()

            buffer.present(view.currentDrawable!)
            buffer.commit()
        }
    }

    // finish by defining the three methods that are required by `NSViewRepresentable` conformance...

    func makeCoordinator() -> Coordinator { Coordinator() }

    func makeNSView(context: NSViewRepresentableContext<MetalView>) -> MTKView {

        let view = MTKView()

        view.delegate = context.coordinator
        view.device = context.coordinator.device
        view.colorPixelFormat = MetalView.pixelFormat
        view.clearColor = MetalView.clearColor
        view.drawableSize = view.frame.size
        view.preferredFramesPerSecond = 60
        view.enableSetNeedsDisplay = false
        view.needsDisplay = true
        view.isPaused = false

        return view
    }

    func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) { }
}

The shader boilerplate is simple. The vert function returns the vertices we pass into it, and the frag function just outputs purple, filling the viewport.

#include <metal_stdlib>

using namespace metal;

vertex float4 init(const device float2 * vertices[[buffer(0)]], const uint vid[[vertex_id]]) {

    return float4(vertices[vid], 0, 1);
}

fragment float4 draw() {

    return float4(0.5, 0, 0.5, 1);
}

In practice, you'll want to create more buffers (like the vertex buffer), and use them to pass data to the GPU (by pointer). You can easily populate the buffers with primitives (including SIMD types). Beyond that, you can create a bridging header, which allows you to define structs in a (separate) C header file, and instantiate them in Metal code and Swift. You can then put any data you want in the buffers.

Note: You should also look into using a semaphore to prevent race cases between the CPU and GPU threads.

Note: There are tutorials (mostly on Apple Developer and Medium) that cover all of these things in enough depth to figure the rest out from the official API docs.

One more thing: Looking through the docs, I noticed that the NSViewRepresentableContext<Self> type has an alias:

typealias Context = NSViewRepresentableContext<Self>

So, you can simplify the signatures for makeNSView and updateNSView. For example, this...

func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) { }

...can be written like this:

func updateNSView(_ nsView: MTKView, context: Context) { }