Showing a MTLTexture on an Entity in RealityKit

Hello @Bersaelor,

Creating a spatial drawing app with RealityKit uses LowLevelTexture for per-frame updates in the "SplashScreenBackgroundComponent".

If you already have a MTLTexture, you can blit its contents to a LowLevelTexture.

-- Greg

Mhmm, in my first simple Test I tried:

@MainActor
private static func generateTexture(width: Int, height: Int) throws -> LowLevelTexture {
    return try LowLevelTexture(descriptor: .init(pixelFormat: .rgba8Unorm_srgb,
                                                 width: width,
                                                 height: height,
                                                 depth: 1,
                                                 mipmapLevelCount: 1,
                                                 textureUsage: [.shaderWrite, .shaderRead]))
}


@MainActor
init(textureSize: SIMD2<Int>) async throws {
    lowLevelTexture = try Self.generateTexture(width: textureSize.x, height: textureSize.y)

    let textureResource = try await TextureResource(from: lowLevelTexture)

    var descriptor = UnlitMaterial.Program.Descriptor()
    descriptor.blendMode = .add

    let program = await UnlitMaterial.Program(descriptor: descriptor)
    material = UnlitMaterial(program: program)
    material.color = .init(texture: .init(textureResource))
    material.opacityThreshold = 0.0 // Enable transparency
    material.blending = .transparent(opacity: 1.0)
}

@MainActor
mutating func setTextureSize(_ textureSize: SIMD2<Int>) throws {
    lowLevelTexture = try Self.generateTexture(width: textureSize.x, height: textureSize.y)
    let textureResource = try TextureResource(from: lowLevelTexture)
    material.color = .init(texture: .init(textureResource))
}

mutating func blitMTLTextureIntoLowLevelTexture(_ mtlTexture: MTLTexture) {
    let size = self.textureSize
    guard mtlTexture.width == size.x, mtlTexture.height == size.y else {
        Logger.ar.error("MTLTexture size \(mtlTexture.width)x\(mtlTexture.height) does not match LowLevelTexture size \(size.x)x\(size.y)")
        return
    }

    MetalHelper.blitTextures(from: mtlTexture, to: lowLevelTexture)
}

And then the blit method:

    static func blitTextures(from inTexture: MTLTexture, to lowLevelTexture: LowLevelTexture) {
        guard let commandQueue = sharedCommandQueue else {
            Logger.ml.error("Failed to get command queue")
            return
        }

        guard let commandBuffer = commandQueue.makeCommandBuffer() else {
            Logger.ml.error("Failed to create command buffer")
            return
        }

        guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {
            Logger.ml.error("Failed to create compute encoder")
            return
        }

        commandBuffer.enqueue()
        defer {
            blitEncoder.endEncoding()
            commandBuffer.commit()
        }

        let outTexture: MTLTexture = lowLevelTexture.replace(using: commandBuffer)

        blitEncoder.copy(from: inTexture, to: outTexture)
    }

which compiles and runs without error, but I only see a pink mesh.

@Bersaelor,

Nothing is jumping out at me in those code snippets as problematic. Can you provide a link to a focused sample project?

--Greg

@DTS Engineer thank you for your reply Greg!

I created a sample app so I can focus on the metal blit outside of the whole AR setup. After taking tiny steps from one thing that worked to the next, I think the blitting now works correctly.

Here's a link the sample application on github

Great, glad to hear that worked out! Creating focused samples is a powerful debugging tool :)

--Greg