Creating and applying a shader to change an entity's rendering in RealityKit

Question

peggers123 OP

Created Oct ’24

Replies 3

Boosts 0

Participants 2

Hello,

I am looking to create a shader to update an entity's rendering. As a basic example say I want to recolour an entity, but leave its original textures showing through:

Screenshot 2024-10-17 at 15.02.39.png

I understand with VisionOS I need to use Reality Composer Pro to create the shader, but I'm lost as how to reference the original colour that I'm trying to update in the node graph. All my attempts appear to completely override the textures in the entity (and its sub-entities) that I want to impact. Also the tutorials / examples I've looked at appear to create materials, not add an effect on top of existing materials.

Any hints or pointers?

Assuming this is possible, I've been trying to load the material in code, and apply to an entity. But do I need to do this to all child entities, or just the topmost?

do {
    let entity = MyAssets.createModelEntity(.plane) // Loads from bundle and performs config
    let material = try await ShaderGraphMaterial(named: "/Root/TestMaterial", from: "Test", in: realityKitContentBundle)
    entity.applyToChildren {
        $0.components[ModelComponent.self]?.materials = [material] 
    }
    root.addChild(entity)
} catch {
    fatalError(error.localizedDescription)
}

Boost

Answer 1

Vision Pro Engineer OP

Apple

Oct ’24

Hi @peggers123

You can recolor an entity without removing its textures by editing the custom Shader Graph material. The key is to multiply the texture color in the shader by your new color. Let me walk you through how you can do this with the biplane model, like you show in your post.

In Reality Composer Pro:

Add the "Toy Biplane" model to your scene.
Open the biplane's Shader Graph material. I believe it's named "usdpreviewsurface3sg".
Locate the "file5_mtlx" Image node and make note of how its output is connected to the Diffuse Color input of the Preview Surface node. You want to intercept this output and multiply it by your new color before it connects to the Diffuse Color input.
So, add a Multiply node.
Connect the Out parameter of the "file5_mtlx" Image node to the top input of the Multiply node, and connect the output of the Multiply node to the Diffuse Color input of the Preview Surface node.
Add a new Uniform Input to the material of type Color3 (Float) and connect it to the bottom of the Multiply node. Name this input "Color".
Change the value of the Color input to recolor the model.

You can also update the color of the model programmatically In Xcode by obtaining a reference to the ShaderGraphMaterial and setting its Color parameter:

RealityView { content in
    // Add the initial RealityKit content
    if let immersiveContentEntity = try? await Entity(named: "Immersive", in: realityKitContentBundle) {
        
        // Get the biplane entity and its Shader Graph material.
        if var biplaneEntity = immersiveContentEntity.findEntity(named: "toy_biplane_base_realistic_lod0"),
           var biplaneMaterial = biplaneEntity.components[ModelComponent.self]?.materials[0] as? ShaderGraphMaterial {
            // Set the color parameter.
            try? biplaneMaterial.setParameter(name: "Color", value: .color(.red))
            // Re-apply the material.
            biplaneEntity.components[ModelComponent.self]?.materials[0] = biplaneMaterial
        }
        
        content.add(immersiveContentEntity)
    }
}

This code assumes you added the biplane model to the default "Immersive" scene that comes with your project, but you can adapt it to load a different model in a different scene if needed.

0

Answer 2

peggers123 OP

Oct ’24

Thank you for your reply. I've replicated these steps and have the effect working as described.

Could I ask a clarification though...

This approach requires each model to have its shader graph updated in Reality Composer Pro to give it this capability. I understand that I could create a re-usable node graph if I wanted to have a more complex effect (the colouring was just a simple effect for sake of the question) - yet still each entity would need updating in Reality Composer Pro to wire it in.

Thus, just to be sure, there is no way to create a shader that can be applied to any model entity. I see RealityKit provides CustomMaterial, which can be setup with a shader, but this is not available in VisionOS at present?

CustomMaterial

0

Answer 3

Vision Pro Engineer OP

Apple

Oct ’24

Hi @peggers123

To clarify, it is possible to create a reusable shader that can be applied to any model.

To do this, I would suggest creating a single shader graph material that supports everything you need to render your models, such as texture mapping, normal mapping, recoloring, and so on. The material applied to the Toy Biplane model is a great foundation to start from.

Next, instead of having each of the Image nodes directly reference their corresponding image file asset (e.g. base color map, normal map, roughness map, and so on), add the image files as uniform inputs to the shader itself and connect them to the Image nodes. This allows you to change which textures a model uses without having to go in and modify the actual shader graph nodes.

Finally, create an instance of this material (secondary click > Create Instance) for each of the models in your scene. Any changes made to the internal logic of the original material will propagate to all of its instances, but each instance can have unique values for its inputs. This means that you can change the image file inputs on a model-to-model basis so that each model can use different textures and shader parameters while still sharing the same underlying shader logic.

Let me know if you'd like me to clarify further!

0