Create MTLLibrary from raw String for use within RealityKit

Hello,

I have a usecase where I need to to download and compile metal shaders on demand as strings or .metal files. These should then be used for CustomMaterials and/or postprocessing within RealityKit. Essentially this boils down to having raw source code that needs to be compiled at runtime.

My plan was to use the method makeLibrary(source:options:completionHandler:) to accomplish this.

The problem is that I get the following error during compilation:

RealityKitARExperienceAssetProvider: An error occured while trying to compile shader library »testShaderLibrary«

- Error Domain=MTLLibraryErrorDomain Code=3 "program_source:2:10: fatal error: 'RealityKit/RealityKit.h' file not found

#include <RealityKit/RealityKit.h>

My code for creating the library looks like this (simplified example):

let librarySourceString: String = """
#include <metal_stdlib>
#include <RealityKit/RealityKit.h>

using namespace metal;

[[visible]]
void mySurfaceShader(realitykit::surface_parameters params)
{
    params.surface().set_base_color(half3(1, 1, 1));
}
"""

mtlDevice.makeLibrary(source: librarySourceString, options: nil) { library, error in
    if let error = error {
        dump(error)
        return
    }

    // do something with library
}

So I'm wondering if there's a way to tell the metal compiler how to resolve this reference to the RealityKit header file? Would I need to replace that part of the source string maybe with an absolute path to the RealityKit framework (if so – how would I get this at runtime)?

Appreciate any hints - thanks!

Replies

Hi, the runtime metal compiler is unable to resolve header dependencies at runtime. The only other option would be to inline/embed the contents of RealityKit/RealityKit.h in your shader source. However, the Metal team does not recommend this since it can lead to UI hitches, you should try to compile your shaders offline if possible. You should be able to find the source in your SDK by command-clicking on a metal file containing #include <RealityKit/RealityKit.h> and choosing 'jump to definition'. Also, please file a feature request to the Metal team for this on Feedback Assistant. Include some info about your use case and post the feedback number here and I can send it to Metal directly.

  • Hi, thanks for your reply! I figured that might be the issue but had hoped there would be another way. That being said, I've tried to inline the header file contents and it actually works really well. As I can prepare the shaders before showing the content anyway, hitches shouldn't be too much of an issue.

    I'll file a feature request tomorrow and post the number here, thanks!

  • Oh btw: if I would want to create an on-device shader editor for geometry modifiers or surface shaders, using the above method is currently the only option – correct? Or is there a way to take some metal files/source strings, compile them and write everything to a file? Is the newish MTLDynamicLibrary API (https://developer.apple.com/documentation/metal/mtldynamiclibrary/3553982-serialize) suitable for this?

  • Yes, you'll have to do the above method for an on-device shader editor. You can certainly try to store your RK custom shaders in a dynamic library, but I would be surprised if that worked since we did not plan for that use case. But do keep us updated, and I'm glad the header inlining worked for you!