RealityKit

RSS for tag

Simulate and render 3D content for use in your augmented reality apps using RealityKit.

Posts under RealityKit tag

200 Posts

Post

Replies

Boosts

Views

Activity

mipmapsMode trade-off?
I am building a 360 photo viewer in VisionOS 26. Which allows the user to choose a 2 by 1 jpg and then renders it with a sphere mesh entity. And I use: TextureResource(contentsOf: url, options: options). I noticed two situations here in terms of mipmaps options. When setting "mipmapsMode: .none": The graphic quality within the "gaze area" looks sharp and clear The two poles (top and bottom) are perfectly rendered Massive shimmer around the "gaze area" When setting "mipmapsMode: .allocateAndGenerateAll": The graphic looks slightly blurrier than in ".none" within the "gaze area" The two poles are very blurry and hard to recognize the texture Much less shimmer around the "gaze area" My question would be: Is there a way to have the perfect graphic quality in ".none" without the massive shimmer? Thank you! Screenshots: mipmapsMode: .none mipmapsMode: .allocateAndGenerateAll
0
0
237
Oct ’25
ARKit Body Tracking not detecting ARBodyAnchor on iOS 26.x (FB15128723)
Since updating to iOS 26.0 (and confirmed on 26.1), ARBodyTrackingConfiguration no longer detects a valid ARBodyAnchor on devices with LiDAR (e.g., iPhone 15 Pro, iPhone 17 Pro Max). This issue reproduces in custom projects and Apple’s official sample “Capturing Body Motion in 3D”. The AR session runs normally, but the delegate call: func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) never yields an ARBodyAnchor with valid joint transforms. All joints return nil when calling: body.skeleton.modelTransform(for: jointName) resulting in 0 valid joints per frame. Environment • Device: iPhone 17 Pro Max (LiDAR) • iOS: 26.0 / 26.1 • Xcode: 16.0 (stable) • Framework: ARKit + RealityKit • Configuration used: config.worldAlignment = .gravityAndHeading config.isAutoFocusEnabled = true config.environmentTexturing = .none session.run(config) Also tested: with and without frameSemantics = .bodyDetection Expected Behavior ARBodyAnchor should be detected and body.skeleton should contain ~89 valid joints with continuous updates.
4
1
1.2k
Dec ’25
SpriteKit/RealityKit + SwiftUI Performance Regression on iOS 26
Hi, Toggling a SwiftUI menu in iOS 26 significantly reduces the framerate of an underlying SKView or ARView. Below are test cases for SpriteKit and RealityKit. I ran these tests on iOS 26.1 Beta using an iPhone 13 (A15 chip). Results were similar on iOS 26.0.1. Both scenes consist of circles and balls bouncing on the ground. The restitution of the physics bodies is set for near-perfect elasticity, so they keep bouncing indefinitely. In both SKView and ARView, the framerate drops significantly whenever the SwiftUI menu is toggled. The menu itself is simple and uses standard SwiftUI animations and styling. SpriteKit import SpriteKit import SwiftUI class SKRestitutionScene: SKScene { override func didMove(to view: SKView) { view.contentMode = .center size = view.bounds.size scaleMode = .resizeFill backgroundColor = .darkGray anchorPoint = CGPoint(x: 0.5, y: 0.5) let groundWidth: CGFloat = 300 let ground = SKSpriteNode(color: .gray, size: CGSize(width: groundWidth, height: 10)) ground.physicsBody = SKPhysicsBody(rectangleOf: ground.size) ground.physicsBody?.isDynamic = false addChild(ground) let circleCount = 5 let spacing: CGFloat = 60 let totalWidth = CGFloat(circleCount - 1) * spacing let startX = -totalWidth / 2 for i in 0..<circleCount { let circle = SKShapeNode(circleOfRadius: 18) circle.fillColor = .systemOrange circle.lineWidth = 0 circle.physicsBody = SKPhysicsBody(circleOfRadius: 18) circle.physicsBody?.restitution = 1 circle.physicsBody?.linearDamping = 0 let x = startX + CGFloat(i) * spacing circle.position = CGPoint(x: x, y: 150) addChild(circle) } } override func willMove(from view: SKView) { self.removeAllChildren() } } struct SKRestitutionView: View { var body: some View { ZStack { SpriteView(scene: SKRestitutionScene(), preferredFramesPerSecond: 120) .ignoresSafeArea() VStack { Spacer() Menu { Button("Edit", systemImage: "pencil") {} Button("Share", systemImage: "square.and.arrow.up") {} Button("Delete", systemImage: "trash") {} } label: { Text("Menu") } .buttonStyle(.glass) } .padding() } } } #Preview { SKRestitutionView() } RealityKit import RealityKit import SwiftUI struct ARViewPhysicsRestitution: UIViewRepresentable { let arView = ARView() func makeUIView(context: Context) -> some ARView { arView.contentMode = .center arView.cameraMode = .nonAR arView.automaticallyConfigureSession = false arView.environment.background = .color(.gray) // MARK: Root let anchor = AnchorEntity() arView.scene.addAnchor(anchor) // MARK: Camera let camera = Entity() camera.components.set(PerspectiveCameraComponent()) camera.position = [0, 1, 4] camera.look(at: .zero, from: camera.position, relativeTo: nil) anchor.addChild(camera) // MARK: Ground let groundWidth: Float = 3.0 let ground = Entity() let groundMesh = MeshResource.generateBox(width: groundWidth, height: 0.1, depth: groundWidth) let groundModel = ModelComponent(mesh: groundMesh, materials: [SimpleMaterial(color: .white, roughness: 1, isMetallic: false)]) ground.components.set(groundModel) let groundShape = ShapeResource.generateBox(width: groundWidth, height: 0.1, depth: groundWidth) let groundCollision = CollisionComponent(shapes: [groundShape]) ground.components.set(groundCollision) let groundPhysicsBody = PhysicsBodyComponent( material: PhysicsMaterialResource.generate(friction: 0, restitution: 0.97), mode: .static ) ground.components.set(groundPhysicsBody) anchor.addChild(ground) // MARK: Balls let ballCount = 5 let spacing: Float = 0.4 let totalWidth = Float(ballCount - 1) * spacing let startX = -totalWidth / 2 let radius: Float = 0.12 let ballMesh = MeshResource.generateSphere(radius: radius) let ballMaterial = SimpleMaterial(color: .systemOrange, roughness: 1, isMetallic: false) let ballShape = ShapeResource.generateSphere(radius: radius) for i in 0..<ballCount { let ball = Entity() let ballModel = ModelComponent(mesh: ballMesh, materials: [ballMaterial]) ball.components.set(ballModel) let ballCollision = CollisionComponent(shapes: [ballShape]) ball.components.set(ballCollision) var ballPhysicsBody = PhysicsBodyComponent( material: PhysicsMaterialResource.generate(friction: 0, restitution: 0.97), /// 0.97 for near perfect elasticity mode: .dynamic ) ballPhysicsBody.linearDamping = 0 ballPhysicsBody.angularDamping = 0 ball.components.set(ballPhysicsBody) let shadow = GroundingShadowComponent(castsShadow: true) ball.components.set(shadow) let x = startX + Float(i) * spacing ball.position = [x, 1, 0] anchor.addChild(ball) } return arView } func updateUIView(_ uiView: UIViewType, context: Context) { } } struct PhysicsRestitutionView: View { var body: some View { ZStack { ARViewPhysicsRestitution() .ignoresSafeArea() .background(.black) VStack { Spacer() Menu { Button("Edit", systemImage: "pencil") {} Button("Share", systemImage: "square.and.arrow.up") {} Button("Delete", systemImage: "trash") {} } label: { Text("Menu") } .buttonStyle(.glass) } .padding() } } } #Preview { PhysicsRestitutionView() }
2
0
275
Oct ’25
Custom Material half visible..?
I'm currently implementing 180° / 360° immersive video for my app. I easily implemented 360° by just applying VideoMaterial to flipped sphere. But I'm stuck at 180°. I'm trying to implement by applying VideoMaterial to hemisphere (half sphere). I want to make VideoMaterial to be visible half front sphere and half back sphere transparent / clear. Would there be any advice / information / idea to implement this? Your help would be grateful.
0
0
140
Oct ’25
Persistent Entity Position
I want to let users place 2D/3D “artworks” on detected walls and have them reappear in exactly the same real‑world spot after quitting and relaunching the app (like widgets do, but for my own entities).Environment: Xcode 26, visionOS 2.0, RealityKit + ARKitSession/WorldTrackingProvider Entities are parented to a holder that’s aligned to a wall via plane/mesh raycasts. What I’ve tried: Create a WorldAnchor at placement, save UUID + full 4×4 transform On next launch, re-create the WorldAnchor (or set the saved transform) and attach the entity Gate restore on relocalization/mesh updates and disable all raycast/search after restore Issue: After relaunch, placement still resolves relative to current device pose, not the same wall position. Questions: Is there a public API in visionOS 2.0 to persist app‑managed world anchors across sessions (room‑fixed), e.g., AnchorStore or equivalent? If not, what’s the recommended pattern to reliably restore wall‑anchored content? Are persistence features mentioned for widgets/windows available to third‑party RealityKit entities?
1
0
256
Oct ’25
Game Controller Input Limitations in visionOS Volumetric Windows - Need Clarification
Game Controller Input Limitations in visionOS Volumetric Windows Hello Apple Developer Community, I'm developing a game for visionOS and have encountered significant limitations with game controller input when using volumetric windows (WindowGroup with .volumetric style). I'd appreciate clarification on whether this is expected behavior and any guidance on best practices. 🧩 Issue Summary When using a DualSense controller with a volumetric window in visionOS, only a subset of controller inputs are available to the app. The remaining inputs appear to be reserved by the system for UI navigation. ✅ Working Inputs (Volumetric Window) D-Pad (all directions) L3 (left thumbstick button click) R3 (right thumbstick button click) Menu button Options button ❌ Not Working Inputs (Volumetric Window) Left thumbstick analog movement (used for UI scrolling instead) Right thumbstick analog movement (used for UI scrolling instead) Face buttons (Cross, Circle, Square, Triangle / A, B, X, Y) Shoulder buttons (L1, R1) Triggers (L2, R2) Key observation: When moving the left thumbstick in a volumetric window, the window's UI scrolls vertically instead of sending input to my app's GameController handlers. Similarly, face buttons seem to be reserved for system UI interactions. ⚙️ Implementation Details I'm using the standard GameController framework: Connect to controller via GCController.controllers() Access extendedGamepad profile Set up valueChangedHandler and pressedChangedHandler for all inputs Handlers confirmed registered via logging Working inputs (D-Pad, L3, R3) trigger immediately and consistently Non-working inputs (thumbsticks, face buttons) never trigger 🧠 Critical Finding: ImmersiveSpace Works Perfectly When testing the exact same code in an ImmersiveSpace (.mixed immersion style), all controller inputs work perfectly: ✅ Both thumbsticks provide full analog input ✅ All face buttons trigger their handlers ✅ All shoulder buttons and triggers work correctly ✅ 100% success rate with no intermittent issues This suggests the issue isn't with my code, but rather how visionOS handles controller input differently between Volumetric Windows and ImmersiveSpace. 🧪 Test Environment I created a minimal test project (Controller-Playground) to isolate the issue: A simple ControllerTester class that registers all GameController handlers A visual UI showing real-time input state No game logic, RealityKit physics, or other complexity Results In volumetric window: Only D-Pad, L3, R3, Menu, Options work In ImmersiveSpace: All inputs work perfectly This confirms the limitation exists at the visionOS platform level, not in app code. 🧰 Attempted Workarounds I tried the following without success: Setting GCSupportsControllerUserInteraction = false in Info.plist Setting UIRequiresFullScreen = true Changing window styles (.plain, .volumetric) Polling vs. handler-based input approaches Various threading models (MainActor, separate thread) Result: The only way to enable full controller support is to switch to ImmersiveSpace. ❓ Questions for Apple Is this input reservation behavior in volumetric windows intended and documented? Are game controllers expected to have limited functionality in volumetric windows while full functionality is reserved for ImmersiveSpace? Is there a way to request full controller input access in a volumetric window, or is ImmersiveSpace the only option for complete controller support? Where can I find official documentation about controller input differences between window types? Are there any APIs or configuration options to disable system controller shortcuts in volumetric windows? 🎯 Impact This limitation has a significant effect on game design and architecture: Volumetric windows offer a multitasking-friendly, less immersive experience ImmersiveSpace provides full controller support but may be more immersive than some games require Games that only need basic D-Pad and button input can work fine in volumetric windows Games requiring analog sticks or face buttons must currently use ImmersiveSpace It would be very helpful if Apple could clarify or reference existing documentation regarding controller input handling in different visionOS window types. If such documentation doesn't exist yet, it might be valuable to include this information in future developer guides or best-practice documents. 🕹 Current Workaround For now, I'm using: D-Pad for character movement (digital 8-direction) R3 (right stick click) as a substitute for the "X" button This setup allows the game to function within a volumetric window, though full controller support still requires ImmersiveSpace. 📄 Request If this is expected behavior, I may have simply missed the relevant documentation — could you please point me to any existing resources that explain this design? If there isn't one yet, it would be great if future visionOS documentation could: Clearly outline controller input behavior across window types Provide guidance on when to use Volumetric Windows vs. ImmersiveSpace for games Consider adding an API option to request full controller access when appropriate If this is not expected behavior, I'm happy to file a detailed bug report with sample code. 💻 System Information visionOS: Latest Simulator Xcode: Latest version Controller: Sony DualSense Framework: GameController (standard extendedGamepad profile) Test project: Minimal reproducible example available Thank you for any clarification or guidance you can provide. This information would be valuable for many developers working on visionOS games.
1
0
643
Oct ’25
RealityView postProcess effect depth texture
Hello, Question re: iOS RealityView postProcess. I've got a working postProcess kernel and I'd like to add some depth-based effects to it. Theoretically I should be able to just do: encoder.setTexture(context.sourceDepthTexture, index: 1) and then in the kernel: texture2d<float, access::read> depthIn [[texture(1)]] ... outTexture.write(depthIn.read(gid), gid); And I consistently see all black rendered to the view. The postProcess shader works, so that's not the issue. It just seems to not be receiving actual depth information. (If I set a breakpoint at the encoder setTexture step, I can see preview the color texture of the scene, but the context's depthTexture looks like all NaN / blank.) I've looked at all the WWDC samples, but they include ARView for all the depth sample code, which has a different set of configuration options than RealityView. So far I haven't seen anywhere to explicitly tell RealityView "include the depth information". So I'm not sure if I'm missing something there. It appears that there is indeed a depth texture being passed, but it looks blank. Is there a working example somewhere that we can reference?
2
0
654
Nov ’25
Multiply exr lightmap in Reality Composer Pro Shader Graph
I’m trying to use EXR lightmaps to overlay baked lighting on top of a base texture in the RCP Shader Graph. When I multiply an EXR image set to Image(float) with an 8-bit base texture, the output becomes Image(float). I can’t connect that to the BaseColor input on the UnlitSurface node, since it only accepts Color3f. I expected to be able to use a Convert node between the Multiply node and the BaseColor input, but when I do that, the result becomes black and white instead of the expected outcome: the EXR multiplied with the base texture using a baseline value of 1, where values below 1 in the EXR would darken the base texture and values above 1 would brighten it. Is there any documentation on how to properly overlay a 32-bit EXR lightmap in the RCP Shader Graph, or is the black-and-white output from the Convert node a bug?
7
0
1.1k
Jan ’26
visionOS 26 - Rendering Issues related to Transparency
Summary After updating to visionOS 26, we’ve encountered severe transparency rendering issues in RealityKit that did not exist in visionOS 2.6 and earlier. These regressions affect applications that dynamically control scene opacity (via OpacityComponent). Our app renders ultra-realistic apartment environments in real time, where users can walk or teleport inside 3D spaces. When the user moves above a speed threshold, we apply a global transparency effect to prevent physical collisions with real-world objects. Everything worked perfectly in visionOS 2.6 — the problems appeared only after upgrading to 26. Scene Setup Overview The environment consists of multiple USDZ models (e.g., architecture, rooms, furniture). We manage LODs manually for performance (e.g., walls and floors always visible in full-res, while rooms swap between low/high-res versions based on user position and field of view). Transparency is achieved using OpacityComponent, applied dynamically when the user moves. Some meshes (e.g., portals to skyboxes, glass windows) use alpha materials We also use OcclusionMaterials to prevent things to be seen through walls when scene is transparent Observed Behavior by Scenario (I can share a video showing the results of each scenario if needed.) Scenario 1 — Severe Flickering (Root Opacity) Setup: OpacityComponent applied to the root entity NO ModelSortGroupComponent used Symptoms: Strong flickering when transparency is active Triangles within the same mesh render at inconsistent opacity levels Appears as if per-triangle alpha sorting is broken Workaround: Moving the OpacityComponent from the root to each individual USDZ entity removes the per-triangle flicker Pros: No conflicts with portals or alpha materials Scenario 2 — Partially Stable, But Alpha Conflicts Setup: OpacityComponent applied per USDZ entity ModelSortGroupComponent(planarUIAlwaysBehind) applied to portal meshes Other entities have NO ModelSortGroupComponent Symptoms: Frequent alpha blending conflicts: Transparent surfaces behind other transparent surfaces flicker or disappear Example: Wine glasses behind glass doors — sometimes neither is rendered, or only one Even opaque meshes behind glass flicker due to depth buffer confusion Alpha materials sometimes render portals or the real world behind them, ignoring other geometry entirely Analysis: Appears related to internal changes in alpha sorting or depth pre-pass behavior introduced in visionOS 26 Pros: Most stable setup so far Cons: Still unreliable when OpacityComponent is active Scenario 3 — Layer Separation Attempt (Regression) Setup: Same as Scenario 2, but: Entities with alpha materials moved to separate USDZs Explicit ModelSortGroupComponent order set (alpha surfaces rendered last) Symptoms: Transparent surfaces behind other transparent surfaces flicker or disappear Depth is completely broken when there's a large transparent surface Alpha materials sometimes render portals or the real world behind them, ignoring other geometry entirely Workaround Attempt: Re-ordering and further separating models did not solve it Pros: None — this setup makes transparency unusable Conclusion There appears to be a regression in RealityKit’s handling of transparency and sorting in visionOS 26, particularly when: OpacityComponent is applied dynamically, and Scenes rely on multiple overlapping transparent materials. These issues did not exist prior to 26, and the same project (no code changes) behaves correctly on previous versions. Request We’d appreciate any insight or confirmation from Apple engineers regarding: Whether alpha sorting or opacity blending behavior changed in visionOS 26 If there are new recommended practices for combining OpacityComponent with transparent materials If a bug report already exists for this regression Thanks in advance!
0
0
213
Nov ’25
ARFrame.sceneDepth not correctly registered with ARFrame.capturedImage for iPad Pro (6th Gen) for high resolution capture.
Hi team, I believe I’ve found a registration issue between ARFrame.sceneDepth and ARFrame.capturedImage when using high-resolution frame capture on a 2022 iPad Pro (6th gen). When enabling high-resolution capture: if let highResFormat = ARWorldTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing { config.videoFormat = highResFormat } … arView.session.captureHighResolutionFrame { ... } the depth map provided by ARFrame.sceneDepth no longer aligns correctly with the corresponding high-resolution capturedImage. This misalignment results in consistently over-estimated distance measurements in my app (which relies on mapping depth to 2D pixel coordinates). iPad Pro (6th gen): misalignment occurs only when capturing high-resolution frames. iPhone 16 Pro: depth is correctly registered for both standard and high-resolution captures. It appears the camera intrinsics, specifically the FOV, change between the “regular” resolution stream and the high-resolution capture on the iPad. My suspicion is that the depth data continues using the intrinsics of the lower resolution stream, resulting in an unregistered depth-to-RGB mapping. Once I have the iPad in hand again, I will confirm whether camera.intrinsics or FOV differ between the low-res and high-res frames. Is this a known issue with high-resolution frame capture on the 2022 iPad Pro? If not, I’m happy to provide some more thorough sample code. Thanks for your time!
0
0
245
Nov ’25
Skybox for iOS/swiftUI not working
I have tried every combination of suggestions to get a skybox to appear. Using swiftUI, realityKit and iOS. Non immersive environment. Does anyone have code that works to display a skybox. When i use a do/catch loop i get environmentResource not found. I have checked the syntax, ensured the folder is referencing the target, used the same name for the folder as the file, the file is a .hdr (i assume this is supported), i have moved the file folder to the top level - no change.
2
0
619
Dec ’25
visualBounds ignores TextComponents set for Entity. Workarounds?
After adding TextComponents to my Entities on visionOS, I have observed that visualBounds will ignore the TextComponents. Documentation states that it should render a rounded rectangle mesh. These mashes are visible on the device, but not visible in the debugger ("Capture Entity Hierarchy") and ignored by visualBounds. Am I missing something? static func makeDirection(_ direction: Direction) -> Entity { let text = Entity() text.name = direction.rawValue text.setScale(SIMD3(repeating: 5), relativeTo: nil) text.transform.rotation = direction.rotation text.components.set(direction.textComponent) return text } My workaround is to add a disabled ModelEntity and take its bounds 😬
1
0
299
Dec ’25
Help: Compiled Timeline Issues
I have developed a fun living diorama world using Reality Composer Pro and XCode. Everything is as it should be, and it looks/works great ... until it does not. If I seem to make any change to any of the 10 timelines that I am using (all on the same scene, no nested scenes), running the app in simulator, device, and via testflight throws errors around compiled timelines, leading to the black screen of death. Every time I clean and run, the timelines in questions might change. Its very frustrating and impossible to track down. Heres are some examples. AssetLoadRequest failed because asset failed to load '/ (3661553931319769725 Timeline (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/Timeline_779.compiledtimeline)' (failed to register asset) Asset / (13631856135570808851 AnimationLibraryAsset (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/AnimationLibraryAsset_1.compiledanimationlibraryasset) failure: failed to register asset Asset 10430065658338454790 AnimationScene (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/AnimationScene_14.compiledanimationscene failure: failed to register asset I went with recommended fixes of closing RCP > Clean Build Folder > Delete Derrived Date (multiple ways) > Re-Open Xcode > Reset Package Cache > Re-Open RCP via XCode > Make a Change > Save > Clean Build Folder Again > Run. Sometimes it works. Most times it does not. I then found my own little work-around, but its not always working as is literally costing me days of wasted time messing around with this. I will DISABLE all timelines, do the above clean method, rerun with no timelines, and it resolves. Then, turn on timelines ONE BY ONE and run until I get another error. Then, rebuild that timeline and nothing else. This is not sustainable. There must be some better way to do this? Or, perhaps I am doing something wrong? Please help if you can.
3
0
317
Jan ’26
Tracking Hand model with skinned mesh and moving fingers
Hi, I have a hand model that is in FBX and I'm exporting it to USD in Blender. I get a skinned mesh and while I can track the whole hand how do I track each joint and assign it and animate the skinned mesh itself. All my attempts say this is not possible in RealityKit as of now. True?
Replies
0
Boosts
0
Views
136
Activity
Oct ’25
mipmapsMode trade-off?
I am building a 360 photo viewer in VisionOS 26. Which allows the user to choose a 2 by 1 jpg and then renders it with a sphere mesh entity. And I use: TextureResource(contentsOf: url, options: options). I noticed two situations here in terms of mipmaps options. When setting "mipmapsMode: .none": The graphic quality within the "gaze area" looks sharp and clear The two poles (top and bottom) are perfectly rendered Massive shimmer around the "gaze area" When setting "mipmapsMode: .allocateAndGenerateAll": The graphic looks slightly blurrier than in ".none" within the "gaze area" The two poles are very blurry and hard to recognize the texture Much less shimmer around the "gaze area" My question would be: Is there a way to have the perfect graphic quality in ".none" without the massive shimmer? Thank you! Screenshots: mipmapsMode: .none mipmapsMode: .allocateAndGenerateAll
Replies
0
Boosts
0
Views
237
Activity
Oct ’25
ARKit Body Tracking not detecting ARBodyAnchor on iOS 26.x (FB15128723)
Since updating to iOS 26.0 (and confirmed on 26.1), ARBodyTrackingConfiguration no longer detects a valid ARBodyAnchor on devices with LiDAR (e.g., iPhone 15 Pro, iPhone 17 Pro Max). This issue reproduces in custom projects and Apple’s official sample “Capturing Body Motion in 3D”. The AR session runs normally, but the delegate call: func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) never yields an ARBodyAnchor with valid joint transforms. All joints return nil when calling: body.skeleton.modelTransform(for: jointName) resulting in 0 valid joints per frame. Environment • Device: iPhone 17 Pro Max (LiDAR) • iOS: 26.0 / 26.1 • Xcode: 16.0 (stable) • Framework: ARKit + RealityKit • Configuration used: config.worldAlignment = .gravityAndHeading config.isAutoFocusEnabled = true config.environmentTexturing = .none session.run(config) Also tested: with and without frameSemantics = .bodyDetection Expected Behavior ARBodyAnchor should be detected and body.skeleton should contain ~89 valid joints with continuous updates.
Replies
4
Boosts
1
Views
1.2k
Activity
Dec ’25
SpriteKit/RealityKit + SwiftUI Performance Regression on iOS 26
Hi, Toggling a SwiftUI menu in iOS 26 significantly reduces the framerate of an underlying SKView or ARView. Below are test cases for SpriteKit and RealityKit. I ran these tests on iOS 26.1 Beta using an iPhone 13 (A15 chip). Results were similar on iOS 26.0.1. Both scenes consist of circles and balls bouncing on the ground. The restitution of the physics bodies is set for near-perfect elasticity, so they keep bouncing indefinitely. In both SKView and ARView, the framerate drops significantly whenever the SwiftUI menu is toggled. The menu itself is simple and uses standard SwiftUI animations and styling. SpriteKit import SpriteKit import SwiftUI class SKRestitutionScene: SKScene { override func didMove(to view: SKView) { view.contentMode = .center size = view.bounds.size scaleMode = .resizeFill backgroundColor = .darkGray anchorPoint = CGPoint(x: 0.5, y: 0.5) let groundWidth: CGFloat = 300 let ground = SKSpriteNode(color: .gray, size: CGSize(width: groundWidth, height: 10)) ground.physicsBody = SKPhysicsBody(rectangleOf: ground.size) ground.physicsBody?.isDynamic = false addChild(ground) let circleCount = 5 let spacing: CGFloat = 60 let totalWidth = CGFloat(circleCount - 1) * spacing let startX = -totalWidth / 2 for i in 0..<circleCount { let circle = SKShapeNode(circleOfRadius: 18) circle.fillColor = .systemOrange circle.lineWidth = 0 circle.physicsBody = SKPhysicsBody(circleOfRadius: 18) circle.physicsBody?.restitution = 1 circle.physicsBody?.linearDamping = 0 let x = startX + CGFloat(i) * spacing circle.position = CGPoint(x: x, y: 150) addChild(circle) } } override func willMove(from view: SKView) { self.removeAllChildren() } } struct SKRestitutionView: View { var body: some View { ZStack { SpriteView(scene: SKRestitutionScene(), preferredFramesPerSecond: 120) .ignoresSafeArea() VStack { Spacer() Menu { Button("Edit", systemImage: "pencil") {} Button("Share", systemImage: "square.and.arrow.up") {} Button("Delete", systemImage: "trash") {} } label: { Text("Menu") } .buttonStyle(.glass) } .padding() } } } #Preview { SKRestitutionView() } RealityKit import RealityKit import SwiftUI struct ARViewPhysicsRestitution: UIViewRepresentable { let arView = ARView() func makeUIView(context: Context) -> some ARView { arView.contentMode = .center arView.cameraMode = .nonAR arView.automaticallyConfigureSession = false arView.environment.background = .color(.gray) // MARK: Root let anchor = AnchorEntity() arView.scene.addAnchor(anchor) // MARK: Camera let camera = Entity() camera.components.set(PerspectiveCameraComponent()) camera.position = [0, 1, 4] camera.look(at: .zero, from: camera.position, relativeTo: nil) anchor.addChild(camera) // MARK: Ground let groundWidth: Float = 3.0 let ground = Entity() let groundMesh = MeshResource.generateBox(width: groundWidth, height: 0.1, depth: groundWidth) let groundModel = ModelComponent(mesh: groundMesh, materials: [SimpleMaterial(color: .white, roughness: 1, isMetallic: false)]) ground.components.set(groundModel) let groundShape = ShapeResource.generateBox(width: groundWidth, height: 0.1, depth: groundWidth) let groundCollision = CollisionComponent(shapes: [groundShape]) ground.components.set(groundCollision) let groundPhysicsBody = PhysicsBodyComponent( material: PhysicsMaterialResource.generate(friction: 0, restitution: 0.97), mode: .static ) ground.components.set(groundPhysicsBody) anchor.addChild(ground) // MARK: Balls let ballCount = 5 let spacing: Float = 0.4 let totalWidth = Float(ballCount - 1) * spacing let startX = -totalWidth / 2 let radius: Float = 0.12 let ballMesh = MeshResource.generateSphere(radius: radius) let ballMaterial = SimpleMaterial(color: .systemOrange, roughness: 1, isMetallic: false) let ballShape = ShapeResource.generateSphere(radius: radius) for i in 0..<ballCount { let ball = Entity() let ballModel = ModelComponent(mesh: ballMesh, materials: [ballMaterial]) ball.components.set(ballModel) let ballCollision = CollisionComponent(shapes: [ballShape]) ball.components.set(ballCollision) var ballPhysicsBody = PhysicsBodyComponent( material: PhysicsMaterialResource.generate(friction: 0, restitution: 0.97), /// 0.97 for near perfect elasticity mode: .dynamic ) ballPhysicsBody.linearDamping = 0 ballPhysicsBody.angularDamping = 0 ball.components.set(ballPhysicsBody) let shadow = GroundingShadowComponent(castsShadow: true) ball.components.set(shadow) let x = startX + Float(i) * spacing ball.position = [x, 1, 0] anchor.addChild(ball) } return arView } func updateUIView(_ uiView: UIViewType, context: Context) { } } struct PhysicsRestitutionView: View { var body: some View { ZStack { ARViewPhysicsRestitution() .ignoresSafeArea() .background(.black) VStack { Spacer() Menu { Button("Edit", systemImage: "pencil") {} Button("Share", systemImage: "square.and.arrow.up") {} Button("Delete", systemImage: "trash") {} } label: { Text("Menu") } .buttonStyle(.glass) } .padding() } } } #Preview { PhysicsRestitutionView() }
Replies
2
Boosts
0
Views
275
Activity
Oct ’25
Widgets For VisionOS26
I found that unofficial apps like ChatGPT and Shadowrocket can use widgets in VisionOS 26. How is this achieved? How can I also enable widgets for apps I develop?
Replies
0
Boosts
0
Views
546
Activity
Oct ’25
Custom Material half visible..?
I'm currently implementing 180° / 360° immersive video for my app. I easily implemented 360° by just applying VideoMaterial to flipped sphere. But I'm stuck at 180°. I'm trying to implement by applying VideoMaterial to hemisphere (half sphere). I want to make VideoMaterial to be visible half front sphere and half back sphere transparent / clear. Would there be any advice / information / idea to implement this? Your help would be grateful.
Replies
0
Boosts
0
Views
140
Activity
Oct ’25
Can the event of long pressing the digital display in VisionPro be monitored in RealityView
Can the event of long pressing the digital display in VisionPro be monitored in RealityView
Replies
1
Boosts
0
Views
74
Activity
Oct ’25
I used colorpick on the view, but after converting it to a entity, colorpick doesn't work. Is there any way to use colorpick
I used colorpick on the view, but after converting it to a model, colorpick doesn't work. Is there any way to use colorpick
Replies
0
Boosts
0
Views
81
Activity
Oct ’25
Persistent Entity Position
I want to let users place 2D/3D “artworks” on detected walls and have them reappear in exactly the same real‑world spot after quitting and relaunching the app (like widgets do, but for my own entities).Environment: Xcode 26, visionOS 2.0, RealityKit + ARKitSession/WorldTrackingProvider Entities are parented to a holder that’s aligned to a wall via plane/mesh raycasts. What I’ve tried: Create a WorldAnchor at placement, save UUID + full 4×4 transform On next launch, re-create the WorldAnchor (or set the saved transform) and attach the entity Gate restore on relocalization/mesh updates and disable all raycast/search after restore Issue: After relaunch, placement still resolves relative to current device pose, not the same wall position. Questions: Is there a public API in visionOS 2.0 to persist app‑managed world anchors across sessions (room‑fixed), e.g., AnchorStore or equivalent? If not, what’s the recommended pattern to reliably restore wall‑anchored content? Are persistence features mentioned for widgets/windows available to third‑party RealityKit entities?
Replies
1
Boosts
0
Views
256
Activity
Oct ’25
Assigning ManipulationComponent to Entity triggers SceneEvents.WillRemoveEntity
When assigning a ManipulationComponent to an Entity SceneEvents.WillRemoveEntity will be called for that Entity. Expected Behavior: the Entity is not (even if temporarily) removed from the Scene and no SceneEvents will be triggered as a result of assigning a ManipulationComponent. FB20872220
Replies
0
Boosts
1
Views
231
Activity
Oct ’25
Game Controller Input Limitations in visionOS Volumetric Windows - Need Clarification
Game Controller Input Limitations in visionOS Volumetric Windows Hello Apple Developer Community, I'm developing a game for visionOS and have encountered significant limitations with game controller input when using volumetric windows (WindowGroup with .volumetric style). I'd appreciate clarification on whether this is expected behavior and any guidance on best practices. 🧩 Issue Summary When using a DualSense controller with a volumetric window in visionOS, only a subset of controller inputs are available to the app. The remaining inputs appear to be reserved by the system for UI navigation. ✅ Working Inputs (Volumetric Window) D-Pad (all directions) L3 (left thumbstick button click) R3 (right thumbstick button click) Menu button Options button ❌ Not Working Inputs (Volumetric Window) Left thumbstick analog movement (used for UI scrolling instead) Right thumbstick analog movement (used for UI scrolling instead) Face buttons (Cross, Circle, Square, Triangle / A, B, X, Y) Shoulder buttons (L1, R1) Triggers (L2, R2) Key observation: When moving the left thumbstick in a volumetric window, the window's UI scrolls vertically instead of sending input to my app's GameController handlers. Similarly, face buttons seem to be reserved for system UI interactions. ⚙️ Implementation Details I'm using the standard GameController framework: Connect to controller via GCController.controllers() Access extendedGamepad profile Set up valueChangedHandler and pressedChangedHandler for all inputs Handlers confirmed registered via logging Working inputs (D-Pad, L3, R3) trigger immediately and consistently Non-working inputs (thumbsticks, face buttons) never trigger 🧠 Critical Finding: ImmersiveSpace Works Perfectly When testing the exact same code in an ImmersiveSpace (.mixed immersion style), all controller inputs work perfectly: ✅ Both thumbsticks provide full analog input ✅ All face buttons trigger their handlers ✅ All shoulder buttons and triggers work correctly ✅ 100% success rate with no intermittent issues This suggests the issue isn't with my code, but rather how visionOS handles controller input differently between Volumetric Windows and ImmersiveSpace. 🧪 Test Environment I created a minimal test project (Controller-Playground) to isolate the issue: A simple ControllerTester class that registers all GameController handlers A visual UI showing real-time input state No game logic, RealityKit physics, or other complexity Results In volumetric window: Only D-Pad, L3, R3, Menu, Options work In ImmersiveSpace: All inputs work perfectly This confirms the limitation exists at the visionOS platform level, not in app code. 🧰 Attempted Workarounds I tried the following without success: Setting GCSupportsControllerUserInteraction = false in Info.plist Setting UIRequiresFullScreen = true Changing window styles (.plain, .volumetric) Polling vs. handler-based input approaches Various threading models (MainActor, separate thread) Result: The only way to enable full controller support is to switch to ImmersiveSpace. ❓ Questions for Apple Is this input reservation behavior in volumetric windows intended and documented? Are game controllers expected to have limited functionality in volumetric windows while full functionality is reserved for ImmersiveSpace? Is there a way to request full controller input access in a volumetric window, or is ImmersiveSpace the only option for complete controller support? Where can I find official documentation about controller input differences between window types? Are there any APIs or configuration options to disable system controller shortcuts in volumetric windows? 🎯 Impact This limitation has a significant effect on game design and architecture: Volumetric windows offer a multitasking-friendly, less immersive experience ImmersiveSpace provides full controller support but may be more immersive than some games require Games that only need basic D-Pad and button input can work fine in volumetric windows Games requiring analog sticks or face buttons must currently use ImmersiveSpace It would be very helpful if Apple could clarify or reference existing documentation regarding controller input handling in different visionOS window types. If such documentation doesn't exist yet, it might be valuable to include this information in future developer guides or best-practice documents. 🕹 Current Workaround For now, I'm using: D-Pad for character movement (digital 8-direction) R3 (right stick click) as a substitute for the "X" button This setup allows the game to function within a volumetric window, though full controller support still requires ImmersiveSpace. 📄 Request If this is expected behavior, I may have simply missed the relevant documentation — could you please point me to any existing resources that explain this design? If there isn't one yet, it would be great if future visionOS documentation could: Clearly outline controller input behavior across window types Provide guidance on when to use Volumetric Windows vs. ImmersiveSpace for games Consider adding an API option to request full controller access when appropriate If this is not expected behavior, I'm happy to file a detailed bug report with sample code. 💻 System Information visionOS: Latest Simulator Xcode: Latest version Controller: Sony DualSense Framework: GameController (standard extendedGamepad profile) Test project: Minimal reproducible example available Thank you for any clarification or guidance you can provide. This information would be valuable for many developers working on visionOS games.
Replies
1
Boosts
0
Views
643
Activity
Oct ’25
RealityView postProcess effect depth texture
Hello, Question re: iOS RealityView postProcess. I've got a working postProcess kernel and I'd like to add some depth-based effects to it. Theoretically I should be able to just do: encoder.setTexture(context.sourceDepthTexture, index: 1) and then in the kernel: texture2d<float, access::read> depthIn [[texture(1)]] ... outTexture.write(depthIn.read(gid), gid); And I consistently see all black rendered to the view. The postProcess shader works, so that's not the issue. It just seems to not be receiving actual depth information. (If I set a breakpoint at the encoder setTexture step, I can see preview the color texture of the scene, but the context's depthTexture looks like all NaN / blank.) I've looked at all the WWDC samples, but they include ARView for all the depth sample code, which has a different set of configuration options than RealityView. So far I haven't seen anywhere to explicitly tell RealityView "include the depth information". So I'm not sure if I'm missing something there. It appears that there is indeed a depth texture being passed, but it looks blank. Is there a working example somewhere that we can reference?
Replies
2
Boosts
0
Views
654
Activity
Nov ’25
Multiply exr lightmap in Reality Composer Pro Shader Graph
I’m trying to use EXR lightmaps to overlay baked lighting on top of a base texture in the RCP Shader Graph. When I multiply an EXR image set to Image(float) with an 8-bit base texture, the output becomes Image(float). I can’t connect that to the BaseColor input on the UnlitSurface node, since it only accepts Color3f. I expected to be able to use a Convert node between the Multiply node and the BaseColor input, but when I do that, the result becomes black and white instead of the expected outcome: the EXR multiplied with the base texture using a baseline value of 1, where values below 1 in the EXR would darken the base texture and values above 1 would brighten it. Is there any documentation on how to properly overlay a 32-bit EXR lightmap in the RCP Shader Graph, or is the black-and-white output from the Convert node a bug?
Replies
7
Boosts
0
Views
1.1k
Activity
Jan ’26
Raytracing on the Vision Pro M5
Is there any support pr plans for support for for raytraced reflections in RealityKit on the Vision Pro M5? I cannot find any documentation regarding this topic.
Replies
2
Boosts
0
Views
502
Activity
Nov ’25
visionOS 26 - Rendering Issues related to Transparency
Summary After updating to visionOS 26, we’ve encountered severe transparency rendering issues in RealityKit that did not exist in visionOS 2.6 and earlier. These regressions affect applications that dynamically control scene opacity (via OpacityComponent). Our app renders ultra-realistic apartment environments in real time, where users can walk or teleport inside 3D spaces. When the user moves above a speed threshold, we apply a global transparency effect to prevent physical collisions with real-world objects. Everything worked perfectly in visionOS 2.6 — the problems appeared only after upgrading to 26. Scene Setup Overview The environment consists of multiple USDZ models (e.g., architecture, rooms, furniture). We manage LODs manually for performance (e.g., walls and floors always visible in full-res, while rooms swap between low/high-res versions based on user position and field of view). Transparency is achieved using OpacityComponent, applied dynamically when the user moves. Some meshes (e.g., portals to skyboxes, glass windows) use alpha materials We also use OcclusionMaterials to prevent things to be seen through walls when scene is transparent Observed Behavior by Scenario (I can share a video showing the results of each scenario if needed.) Scenario 1 — Severe Flickering (Root Opacity) Setup: OpacityComponent applied to the root entity NO ModelSortGroupComponent used Symptoms: Strong flickering when transparency is active Triangles within the same mesh render at inconsistent opacity levels Appears as if per-triangle alpha sorting is broken Workaround: Moving the OpacityComponent from the root to each individual USDZ entity removes the per-triangle flicker Pros: No conflicts with portals or alpha materials Scenario 2 — Partially Stable, But Alpha Conflicts Setup: OpacityComponent applied per USDZ entity ModelSortGroupComponent(planarUIAlwaysBehind) applied to portal meshes Other entities have NO ModelSortGroupComponent Symptoms: Frequent alpha blending conflicts: Transparent surfaces behind other transparent surfaces flicker or disappear Example: Wine glasses behind glass doors — sometimes neither is rendered, or only one Even opaque meshes behind glass flicker due to depth buffer confusion Alpha materials sometimes render portals or the real world behind them, ignoring other geometry entirely Analysis: Appears related to internal changes in alpha sorting or depth pre-pass behavior introduced in visionOS 26 Pros: Most stable setup so far Cons: Still unreliable when OpacityComponent is active Scenario 3 — Layer Separation Attempt (Regression) Setup: Same as Scenario 2, but: Entities with alpha materials moved to separate USDZs Explicit ModelSortGroupComponent order set (alpha surfaces rendered last) Symptoms: Transparent surfaces behind other transparent surfaces flicker or disappear Depth is completely broken when there's a large transparent surface Alpha materials sometimes render portals or the real world behind them, ignoring other geometry entirely Workaround Attempt: Re-ordering and further separating models did not solve it Pros: None — this setup makes transparency unusable Conclusion There appears to be a regression in RealityKit’s handling of transparency and sorting in visionOS 26, particularly when: OpacityComponent is applied dynamically, and Scenes rely on multiple overlapping transparent materials. These issues did not exist prior to 26, and the same project (no code changes) behaves correctly on previous versions. Request We’d appreciate any insight or confirmation from Apple engineers regarding: Whether alpha sorting or opacity blending behavior changed in visionOS 26 If there are new recommended practices for combining OpacityComponent with transparent materials If a bug report already exists for this regression Thanks in advance!
Replies
0
Boosts
0
Views
213
Activity
Nov ’25
Displaying Gaussian splats in visionOS
Apple's new Spatial Personas use Gaussian Splatting, but I have not found any APIs for visionOS to display a Gaussian Splat like a PLY file. Am I just missing the Apple documentation? If not, are there common practices developers are using for displaying Gaussian Splats in visionOS?
Replies
1
Boosts
2
Views
625
Activity
Jan ’26
ARFrame.sceneDepth not correctly registered with ARFrame.capturedImage for iPad Pro (6th Gen) for high resolution capture.
Hi team, I believe I’ve found a registration issue between ARFrame.sceneDepth and ARFrame.capturedImage when using high-resolution frame capture on a 2022 iPad Pro (6th gen). When enabling high-resolution capture: if let highResFormat = ARWorldTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing { config.videoFormat = highResFormat } … arView.session.captureHighResolutionFrame { ... } the depth map provided by ARFrame.sceneDepth no longer aligns correctly with the corresponding high-resolution capturedImage. This misalignment results in consistently over-estimated distance measurements in my app (which relies on mapping depth to 2D pixel coordinates). iPad Pro (6th gen): misalignment occurs only when capturing high-resolution frames. iPhone 16 Pro: depth is correctly registered for both standard and high-resolution captures. It appears the camera intrinsics, specifically the FOV, change between the “regular” resolution stream and the high-resolution capture on the iPad. My suspicion is that the depth data continues using the intrinsics of the lower resolution stream, resulting in an unregistered depth-to-RGB mapping. Once I have the iPad in hand again, I will confirm whether camera.intrinsics or FOV differ between the low-res and high-res frames. Is this a known issue with high-resolution frame capture on the 2022 iPad Pro? If not, I’m happy to provide some more thorough sample code. Thanks for your time!
Replies
0
Boosts
0
Views
245
Activity
Nov ’25
Skybox for iOS/swiftUI not working
I have tried every combination of suggestions to get a skybox to appear. Using swiftUI, realityKit and iOS. Non immersive environment. Does anyone have code that works to display a skybox. When i use a do/catch loop i get environmentResource not found. I have checked the syntax, ensured the folder is referencing the target, used the same name for the folder as the file, the file is a .hdr (i assume this is supported), i have moved the file folder to the top level - no change.
Replies
2
Boosts
0
Views
619
Activity
Dec ’25
visualBounds ignores TextComponents set for Entity. Workarounds?
After adding TextComponents to my Entities on visionOS, I have observed that visualBounds will ignore the TextComponents. Documentation states that it should render a rounded rectangle mesh. These mashes are visible on the device, but not visible in the debugger ("Capture Entity Hierarchy") and ignored by visualBounds. Am I missing something? static func makeDirection(_ direction: Direction) -> Entity { let text = Entity() text.name = direction.rawValue text.setScale(SIMD3(repeating: 5), relativeTo: nil) text.transform.rotation = direction.rotation text.components.set(direction.textComponent) return text } My workaround is to add a disabled ModelEntity and take its bounds 😬
Replies
1
Boosts
0
Views
299
Activity
Dec ’25
Help: Compiled Timeline Issues
I have developed a fun living diorama world using Reality Composer Pro and XCode. Everything is as it should be, and it looks/works great ... until it does not. If I seem to make any change to any of the 10 timelines that I am using (all on the same scene, no nested scenes), running the app in simulator, device, and via testflight throws errors around compiled timelines, leading to the black screen of death. Every time I clean and run, the timelines in questions might change. Its very frustrating and impossible to track down. Heres are some examples. AssetLoadRequest failed because asset failed to load '/ (3661553931319769725 Timeline (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/Timeline_779.compiledtimeline)' (failed to register asset) Asset / (13631856135570808851 AnimationLibraryAsset (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/AnimationLibraryAsset_1.compiledanimationlibraryasset) failure: failed to register asset Asset 10430065658338454790 AnimationScene (RealityFileAsset)URL/file:///var/containers/Bundle/Application/F4408256-6014-4264-9E4B-F74AEF0EDE53/SantasVillage.app/RealityKitContent_RealityKitContent.bundle/RealityKitContent.reality/AnimationScene_14.compiledanimationscene failure: failed to register asset I went with recommended fixes of closing RCP > Clean Build Folder > Delete Derrived Date (multiple ways) > Re-Open Xcode > Reset Package Cache > Re-Open RCP via XCode > Make a Change > Save > Clean Build Folder Again > Run. Sometimes it works. Most times it does not. I then found my own little work-around, but its not always working as is literally costing me days of wasted time messing around with this. I will DISABLE all timelines, do the above clean method, rerun with no timelines, and it resolves. Then, turn on timelines ONE BY ONE and run until I get another error. Then, rebuild that timeline and nothing else. This is not sustainable. There must be some better way to do this? Or, perhaps I am doing something wrong? Please help if you can.
Replies
3
Boosts
0
Views
317
Activity
Jan ’26