How can I swap materials on a specific object within a scene loaded from realitykit? I want to have 4-6 materials to be able to select from and swap out on an object.

In your main view you can use SwiftUI to add buttons to allow the user to tap and change the current material. You can set that selected material onto a ViewModel object, or have the button actions trigger a function on the ViewModel. Present an ImmersiveSpace and use the same ViewModel object over there. Add your desired ModelEntity to the RealityView of the ImmersiveSpace. Make a reference to that ModelEntity in your ViewModel. Back in your SwiftUI View and ViewModel, make your button actions modify the .material of the ModelEntity previously added. Now, you have a SwiftUI view with button actions hooked up to the same ViewModel that has a reference to the ModelEntity.