I'm currently developing an application where the models present inside a volumetric window may exceed the clipping boundaries of the window. ( Which I currently understand to be a maximum of 2m ) Because of this, as models move through the clipping boundaries, the interior of the models becomes visible. If possible, I'd like to cap these interiors with a solid fill so as to make them more visually appealing. However, as far as I can tell, I'm quite limited in how I might be able to achieve this when using RealityKit on VisionOS. Some approaches I've seen to accomplish similar effects seem to use multiple passes of model geometries rendering into stencil buffers and using that to inform whether or not a cap should be drawn. However, afiact, if I have opted into using a RealityView and RealityKit, I don't have the level of control over my render pipeline such that I can render ModelEntities and also have multiple rendering passes over the set of contained entities to render into a stencil buffer that I then provide to a separate set of "capping planes" ( how I currently imagine I might accomplish this effect ). Alternatively ( due to the nature of the models I'm using ) I considered using a height map to construct an approximation of a surface cap, but how I might use a shader to construct a height map of rendered entities seems similarly difficult using the VisionOS RealityView pipeline. It is not obvious to me how I could use a ShaderGraphMaterial to render to an arbitrary image buffer that I might then pass to other functions to use as an input; ShaderGraphMaterial seems biased to the fact that all image inputs and outputs are either literal files or the actual rendered buffer. Would anyone out there have already created an effect like this that might have some advice? Or, potentially correct any misunderstandings I have with regards to accessing the Metal pipeline for RealityView or using ShaderGraphMaterial to construct a height map?

The transparency in reality kit is not rendered properly from specific ordinal axes. It seems like it is a depth sorting issue where it is rejecting some transparent surfaces when it should not. Some view directions relative to specific ordinal axes are fine. I have not narrowed down which specific axes is the problem. This is true across particle systems and/or meshes. It is very easy to replicate this issues using multiple transparent meshes or particle systems. In the above gif you can see the problem in multiple instances, the fire and snow particles are sorted behind the terrain, which has transparency since it is a procedural blend of grass, rock, and ice, but it is correctly sorted in front of the opaque materials such the rocks and wood. In the above gif, it is two back to back grid meshes (since dual sided rendering is not supported) that have a custom surface shader to animate the mesh in a wave and also apply transperency. You can see in the distance, where the transparency seems to be rendered/overlapped correctely, but at the overlap approaches the screen (and crosses an ordinal axes) it renders black for the transparent portion of the surface, when the green of the mesh that is behind should be rendered. This is a blocking problem for the development of this demo.

I'm trying to build a project with a moderately complex Reality Composer Pro project, but am unable to because my Mac mini (2023, 8GB RAM) keeps running out of memory. I'm wondering if there are any known memory leaks in realitytool, but basically the tool is taking up 20-30GB (!) memory during builds. I have a Mac Pro for content creation, which is why I didn't go for more RAM on the mini – it was supposed to just be a build machine for Apple Silicon compatibility, as my Pro is Intel. But, I'm kinda stuck here. I have a scene that builds fine, but any time I had a USD – in this case a tree asset – with lots of instances, or a lot of geometry, I run into the memory issue. I've tried greatly simplifying the model, but even a 2MB USD is resulting in the crash. I'm failing to see how adding a 2MB asset would cause the memory of realitytool to balloon so much during builds. If someone from Apple is willing to look, I can provide the scene – but it's proprietary so I can't just post it publicly here.

I'm trying to get a similar experience to Apple TV's immersive videos, but I cannot figure out how to present the AVPlayerViewController controls detached from the video. I am able to use the same AVPlayer in a window and projected on a VideoMaterial, but I can't figure out how to just present the controls, while displaying the video only in the 3D entity, without having a 2D projection in any view. Is this even possible?

In my project, i want to use new shadergraphmaterial to do the stereoscopic render, i notice that there is a node called Camera Index Switch Node can do this. But when i tried it , i found that : It can only output Integer type value, when i change to float value , it change back again, i don't konw if it is a bug. 2. So i test this node with a IF node，i found that it output is weird. Below is zero should output，it is black but when i change to IF node，it is grey，it is neither 0 nor 1（My IF node result is TRUE result 1, FALSE result 0） I wanna ask if this is a bug, and if this is a correct way to do the stereoscopic render.

Hey, I'm wondering what would be the proper way to add RealityView content asynchronously, while doing the heavy lifting in a background thread. My use case is that I am generating procedural geometry which takes a few seconds to complete. Meanwhile I would like the UI to show other geometry / UI elements and the Main thread to be responsive. Basically what I would like to do, in pseudocode, is: runInBackgroundThread { let geometry = generateGeometry() // CPU intensive, takes 1-2 s let entity = createEntity(geometry) // CPU intensive, takes ~1 s let material = try! await ShaderGraphMaterial(..) entity.model!.materials = [material] runInMainThread { addToRealityViewContent(entity) } } With this I am running into so many issues with especially the material, which apparently cannot be constructed on a non-main thread and cannot be passed over thread borders.

I am trying to make a shader for a disco ball lighting effect for my app. I want the light to reflect on the scene mesh. i was curious if anyone has pointers on how to do this in shader graph in reality composer pro or writing a surface shader. The effect rotates the dots as the ball spins. This is the effect in the apple clips that applies the effect to the scene mesh

Adding AVPlayer as attachment on the side using RealityKit. The video in it thought is not aligned. And thoughts on what could be going wrong? RealityView { content, attachments in let url = self.video.resolvedURL let asset = AVURLAsset(url: url) let playerItem = AVPlayerItem(asset: asset) var videoPlayerComponent = VideoPlayerComponent(avPlayer: player) videoPlayerComponent.isPassthroughTintingEnabled = true // entity.components[VideoPlayerComponent.self] = videoPlayerComponent entity.position = [0, 0, 0] entity.scale *= 0.50 player.replaceCurrentItem(with: playerItem) player.play() content.add(entity) } update: { content, attachments in // if content.entities.count < 2 { if showAnotherPlayer { if let attachment = attachments.entity(for: "Attachment") { playerModel.loadVideo(library.selectedVideo!, presentation: .fullWindow) //4. Position the Attachment and add it to the RealityViewContent attachment.position = [1.0, 0, 0] attachment.scale *= 1.0 //let radians = -45.0 * Float.pi / 180.0 //attachment.transform.rotation += simd_quatf(angle: radians, axis: SIMD3<Float>(0,1,0)) let entity = content.entities.first attachment.setParent(entity) content.add(attachment) } } if showLibrary { if let attachment = attachments.entity(for: "Featured") { //4. Position the Attachment and add it to the RealityViewContent attachment.position = [0.0, -0.3, 0] attachment.scale *= 0.7 //let radians = -45.0 * Float.pi / 180.0 //attachment.transform.rotation += simd_quatf(angle: radians, axis: SIMD3<Float>(0,1,0)) let entity = content.entities.first attachment.setParent(entity) viewModel.attachment = attachment content.add(attachment) } } else { if let scene = content.entities.first?.scene { let _ = print("found scene") } if let featuredEntity = content.entities.first?.scene?.findEntity(named: "Featured") { let _ = print("featured entity found") } if let attachment = viewModel.attachment { let _ = print("-- removing attachment") if let anchor = attachment.anchor { let _ = print("-- removing anchor") anchor.removeFromParent() } attachment.removeFromParent() content.remove(attachment) } else { let _ = print("the attachment is missing") } } // } } attachments: { Attachment(id: "Attachment") { PlayerView() .frame(width: 2048, height: 1024) .environment(library) .environment(playerModel) .onAppear { DispatchQueue.main.asyncAfter(deadline: .now()+1) { playerModel.play() } } .onDisappear { } } if showLibrary { Attachment(id: "Featured") { VideoListView(title: "Featured", videos: library.videos, cardStyle: .full, cardSpacing: 20) { video in library.selectedVideo = video showAnotherPlayer = true } .frame(width: 2048, height: 1024) } } } PlayerView

Hi, I am implementing player using RealityKit's VideoPlayerComponent and AVPlayer. When app enter immersive space, playback beigns. But only audio playabck, I can't see video. Do I need specify entity's position and size? struct MyApp: App { @State private var playerImmersionStyle: ImmersionStyle = .full var body: some Scene { WindowGroup { ContentView() } .defaultSize(width: 800, height: 200) ImmersiveSpace(id: "playerImmersionStyle") { ImmersiveSpaceView() } .immersionStyle(selection: $playerImmersionStyle, in: playerImmersionStyle) } func application(_ application: UIApplication, configurationForConnecting connectingSceneSession: UISceneSession, options: UIScene.ConnectionOptions) -> UISceneConfiguration { return UISceneConfiguration(name: "My Scene Configuration", sessionRole: connectingSceneSession.role) } } struct PlayerViewEx: View { let entity = Entity() var body: some View { RealityView() { content in let entity = makeVideoEntity() content.add(entity) } } public func makeVideoEntity() -> Entity { let url = Bundle.main.url(forResource: "football", withExtension: "mov")! let asset = AVURLAsset(url: url) let playerItem = AVPlayerItem(asset: asset) let player = AVPlayer() var videoPlayerComponent = VideoPlayerComponent(avPlayer: player) videoPlayerComponent.isPassthroughTintingEnabled = true entity.components[VideoPlayerComponent.self] = videoPlayerComponent entity.scale *= 0.4 player.replaceCurrentItem(with: playerItem) player.play() return entity } } #Preview { PlayerViewEx() }

I'm developing an app for Apple Vision Pro and have a question about RealityKit. Recently, I attempted to use drag gestures to manipulate two entities, A and B, with my left and right hands respectively. The two entities belong to the same RealityView. I anticipated that I could move Entity A with my left hand and Entity B with my right hand independently. However, I noticed that the movement of one hand affects both entities simultaneously. Presumably, DragGesture().onChanged is triggered twice for each entity. In an attempt to properly pair each hand with its corresponding entity, I investigated the platform.manipulatorGroup in the debugger. However, I encountered a compile error when trying to access the platform variable. Is it feasible to pair each hand with a specific entity and move both objects separately? Thank you in advance.

We are developing an AR app which uses spatial audio. If we want to use Realitykit to create the app, will we need to use a MacBook Pro running Silicon?

Hello, I've been trying to leverage instanced rendering in RealityKit on visionOS but have not had success. RealityKit states this is supported: https://developer.apple.com/documentation/realitykit/validating-usd-files https://developer.apple.com/videos/play/wwdc2021/10075/?time=1373 https://developer.apple.com/videos/play/wwdc2023/10099/?time=772 RealityKit Trace metrics Validating instancing is working: To test I made a base visionOS app with immersive space and the entity replaced with my test usdz file. I've been using the RealityKit Trace profiling template in xcode instruments in the immersive space and volume closed. This gets consistent draw call results. If I have a single sphere mesh with one material I get one draw call, but the number of draw calls grows linearly with mesh count no matter how my entity is configured. What I've tried Create a test scene in blender, export with instancing enabled Create a test scene in Reality Composer Pro using references Author usda files by hand based on the OpenUSD spec Programatically create a MeshResource with Contents at runtime References https://openusd.org/release/api/_usd__page__scenegraph_instancing.html https://developer.apple.com/documentation/realitykit/meshresource https://developer.apple.com/documentation/realitykit/meshresource/instance Thank you

Is there any way to specify a clip volume or clipping planes on either a RealityView or the underlying RealityKit entity on visionOS? This was easy on SceneKit with shader modifiers, or in OpenGL, or WebGL, or with RealityKit on iOS or macOS with CustomMaterial surface shader, but CustomMaterial is not supported on visionOS.

Hello, I'm currently building an app that implements the on-device object capture API to create 3D models. I have two concerns that I cannot find addressed anywhere on the internet: Can on-device object capture be performed by devices without LiDAR? I understand that depth data is necessary for making scale-accurate models - if there is an option to disable it, where would one specify that in code? Can models be exported to .obj instead of .usdz? From WWDC2021 at 3:00 it is mentioned that it is possible with the Apple Silicon API but what about with on-device scanning? I would be very grateful if anyone is knowledgeable enough to provide some insight. Thank you so much!

I'm trying to better understand how loading entities works. If I do this: RealityView { content in // Add the initial RealityKit content if let scene = try? await Entity(named: "RCP_Scene", in: realityKitContentBundle) { content.add(scene) } } It returns the root with the two objects I have in the scene (sphere_01 and sphere_02). If I add a drag gesture to this entity it works on the root and gets applied to both sphere_01 and sphere_02 together (they both indiviually have collision and input components set to allow gestures). How do I get individual control of sphere_01 and sphere_02? Is it possible to load the root scene, as I'm doing above, and have individual control?

Hi, I am investigating how to emit the following in my visionOS app. https://www.hiroakit.com/archives/1432 https://blog.terresquall.com/2020/01/getting-your-emission-maps-to-work-in-unity/ Right now, I'm trying various things with Shader Graph in Reality Composer Pro, but I can't tell from the official documentation and WWDC session videos what the individual functions and combined effects of Reality Composer Pro's Shader Graph nodes are, I am having a hard time understanding the effects of the individual functions and combinations of them. I have a feeling that such luminous materials and expressions are not possible in visionOS to begin with. If there is a way to achieve this, please let me know. Thanks.

Hi, I'm trying to rotate an entity in VisionPro. Most of the code is the same as the Diorama code from WWDC23. The problem I'm having is that the rotiation occurs but the axis of the rotation is not the center of my object. It seems to be centered on the zero coordinate of the immersive space . How do I change the rotation3DEffect to tell it to rotate around the entity? Not the space? Is it even possible? This is the code, the rotation is at the end. var body: some View { @Bindable var viewModel = viewModel RealityView { content, _ in do { let entity = try await Entity(named: "DioramaAssembled", in: RealityKitContent.RealityKitContentBundle) viewModel.rootEntity = entity content.add(entity) viewModel.updateScale() // Offset the scene so it doesn't appear underneath the user or conflict with the main window. entity.position = SIMD3<Float>(0, 0, -2) subscriptions.append(content.subscribe(to: ComponentEvents.DidAdd.self, componentType: PointOfInterestComponent.self, { event in createLearnMoreView(for: event.entity) })) entity.generateCollisionShapes (recursive: true) entity.components.set(InputTargetComponent()) } catch { print("Error in RealityView's make: \(error)") } } .rotation3DEffect(.radians(currentrotateByX), axis: .y) .rotation3DEffect(.radians(currentrotateByY), axis: .x)

In a RealityView, I have scene loaded from Reality Composer Pro. The entity I'm interacting with has a PhysicallyBasedMaterial with a diffuse color. I want to change that color when on long press. I can get the entity and even get a reference to the material, but I can't seem to change anything about it. What is the best way to change the color of a material at runtime? var longPress: some Gesture { LongPressGesture(minimumDuration: 0.5) .targetedToAnyEntity() .onEnded { value in value.entity.position.y = value.entity.position.y + 0.01 if var shadow = value.entity.components[GroundingShadowComponent.self] { shadow.castsShadow = true value.entity.components.set(shadow) } if let model = value.entity.components[ModelComponent.self] { print("material", model) if let mat = model.materials.first { print("material", mat) // I have a material here but I can't set any properties? // mat.diffuseColor does not exist } } } } Here is the full code struct Lab5026: View { var body: some View { RealityView { content in if let root = try? await Entity(named: "GestureLab", in: realityKitContentBundle) { root.position = [0, -0.45, 0] if let subject = root.findEntity(named: "Cube") { subject.components.set(HoverEffectComponent()) subject.components.set(GroundingShadowComponent(castsShadow: false)) } content.add(root) } } .gesture(longPress.sequenced(before: dragGesture)) } var longPress: some Gesture { LongPressGesture(minimumDuration: 0.5) .targetedToAnyEntity() .onEnded { value in value.entity.position.y = value.entity.position.y + 0.01 if var shadow = value.entity.components[GroundingShadowComponent.self] { shadow.castsShadow = true value.entity.components.set(shadow) } if let model = value.entity.components[ModelComponent.self] { print("material", model) if let mat = model.materials.first { print("material", mat) // I have a material here but I can't set any properties? // mat.diffuseColor does not exist // PhysicallyBasedMaterial } } } } var dragGesture: some Gesture { DragGesture() .targetedToAnyEntity() .onChanged { value in let newPostion = value.convert(value.location3D, from: .global, to: value.entity.parent!) let limit: Float = 0.175 value.entity.position.x = min(max(newPostion.x, -limit), limit) value.entity.position.z = min(max(newPostion.z, -limit), limit) } .onEnded { value in value.entity.position.y = value.entity.position.y - 0.01 if var shadow = value.entity.components[GroundingShadowComponent.self] { shadow.castsShadow = false value.entity.components.set(shadow) } } } }

I have some strange behavior in my app. When I set the position to .zero you can see the sphere normally. But when I change it to any number it doesn't matter which and how small. The Sphere isn't visible or in the view. The RealityView import SwiftUI import RealityKit import RealityKitContent struct TheSphereOfDoomRV: View { @StateObject var viewModel: SphereViewModel = SphereViewModel() let sphere = SphereEntity(radius: 0.25, materials: [SimpleMaterial(color: .red, isMetallic: true)], name: "TheSphere") var body: some View { RealityView { content, attachments in content.add(sphere) } update: { content, attachments in sphere.scale = SIMD3<Float>(x: viewModel.scale, y: viewModel.scale, z: viewModel.scale) } attachments: { VStack { Text("The Sphere of Doom is one of the most powerful Objects. You can interact with him in every way you can imagine ").multilineTextAlignment(.center) Button { } label: { Text("Play Video!") } }.tag("description") }.modifier(GestureModifier()).environmentObject(viewModel) } } SphereEntity: import Foundation import RealityKit import RealityKitContent class SphereEntity: Entity { private let sphere: ModelEntity @MainActor required init() { sphere = ModelEntity() super.init() } init(radius: Float, materials: [Material], name: String) { sphere = ModelEntity(mesh: .generateSphere(radius: radius), materials: materials) sphere.generateCollisionShapes(recursive: false) sphere.components.set(InputTargetComponent()) sphere.components.set(HoverEffectComponent()) sphere.components.set(CollisionComponent(shapes: [.generateSphere(radius: radius)])) sphere.name = name super.init() self.addChild(sphere) self.position = .zero // .init(x: Float, y: Float, z: Float) and [Float, Float, Float] doesn't work ... } }

Does anyone know how I can disable foveation for an ImmersiveSpace? I'm aware that I could use a CompositorLayer and my own Metal rendering to control foveation, but I'm hoping that I can configure an existing/underlying LayerRenderer (or similar) to disable it for an immersive scene. Or if there's another approach I should be taking, any pointers are appreciated. Thank you!