RealityKit

RSS for tag

Simulate and render 3D content for use in your augmented reality apps using RealityKit.

Posts under RealityKit tag

200 Posts

Post

Replies

Boosts

Views

Activity

Crash when Displaying RealityView on Multiple Screen only Connecting with Xcode
I have an iOS app that uses RealityView to display some models and interact with them, and the app uses regular iOS app navigations, then a challenge I'm facing is how to maintain multiple RealityView on multiplescreens. For example Screen A has a RealityView, and then I navigate to Screen B (also has a RealityView) using stack based navigation, when I do so I got a crash -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation Fragment Function(fsRealityPbr): argument envProbeTable[0] from Buffer(7) with offset(0) and length(16) has space for 16 bytes, but argument has a length(864). Fragment Function(fsRealityPbr): incorrect type of texture (MTLTextureType2D) bound at Texture binding at index 20 (expect MTLTextureTypeCubeArray) for envProbeDiffuseArray[0]. Interestingly this crash only happens when debugging with Xcode, not happens when the app runs on its own. I'm not sure what I'm doing is anti-pattern or it's some Xcode debugging limitation.
0
0
749
Mar ’26
Draw An Outline Around a Model Entity
Hi, Is there a resource or sample code about how to draw an outline around a mesh in RealityKit? Typically, this is useful for visualizing a selection, like in Reality Composer Pro. How to achieve such effect? A shader material? A post-process effect in ARView or RealityRenderer? Methods such as duplicating the entity mesh, scaling it, and using material.faceCulling = .front did not look good in my experiments. Thank you.
1
0
543
Feb ’26
Reality Kit Scene
Hi, I’m wondering whether RealityKit has its own scene management system, since it uses ARView (backed by ARKit) to present AR content. Does RealityKit manage scenes independently, or does it rely entirely on ARKit’s scene handling? Thank you.
1
0
156
Feb ’26
Unexpected behavior when writing entities and loading realityFiles.
I have a simple visionOS app that creates an Entity, writes it to the device, and then attempts to load it. However, when the entity file get overwritten, it affects the ability for the app to load it correctly. Here is my code for saving the entity. import SwiftUI import RealityKit import UniformTypeIdentifiers struct ContentView: View { var body: some View { VStack { ToggleImmersiveSpaceButton() Button("Save Entity") { Task { // if let entity = await buildEntityHierarchy(from: urdfPath) { let type = UTType.realityFile let filename = "testing.\(type.preferredFilenameExtension ?? "bin")" let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let fileURL = documentsURL.appendingPathComponent(filename) do { let mesh = MeshResource.generateBox(size: 1, cornerRadius: 0.05) let material = SimpleMaterial(color: .blue, isMetallic: true) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) let entity = Entity() entity.components.set(modelComponent) print("Writing \(fileURL)") try await entity.write(to: fileURL) } catch { print("Failed writing") } } } } .padding() } } Every time I press "Save Entity", I see a warning similar to: Writing file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality Failed to set dependencies on asset 1941054755064863441 because NetworkAssetManager does not have an asset entity for that id. When I open the immersive space, I attempt to load the same file: import SwiftUI import RealityKit import UniformTypeIdentifiers struct ImmersiveView: View { @Environment(AppModel.self) private var appModel var body: some View { RealityView { content in guard let type = UTType.realityFile.preferredFilenameExtension else { return } let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let fileURL = documentsURL.appendingPathComponent("testing.\(type)") guard FileManager.default.fileExists(atPath: fileURL.path) else { print("❌ File does not exist at path: \(fileURL.path)") return } if let entity = try? await Entity(contentsOf: fileURL) { content.add(entity) } } } } I also get errors after I overwrite the entity (by pressing "Save Entity" after I have successfully loaded it once). The warnings that appear when the Immersive space attempts to load the new entity are: Asset 13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'. Asset 8308977590385781534 Scene (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Scene_0.compiledscene failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to read archive entry. AssetLoadRequest failed because asset failed to load '13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh' (Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'.) The order of operations to make this happen: Launch app Press "Save Entity" to save the entity "Open Immersive Space" to view entity Press "Save Entity" to overwrite the entity "Open Immersive Space" to view entity, failed asset load request Also Launch app, the entity should still be save from last time the app ran "Open Immersive Space" to view entity Press "Save Entity" to overwrite the entity "Open Immersive Space" to view entity, failed asset load request NOTE: It appears I can get it to work slightly better by pressing the "Save Entity" button twice before attempting to view it again in the immersive space.
2
3
410
Feb ’26
RealityKit_DirectionalLight_Question
My application calculates three distinct Meesus Double [x, y, z] Radian values to light a sphere in RealityKit with DirectionalLight. It is my understanding that I must use (simd_quatf) for each radian value to properly light the sphere in the view. The code correctly [orientates] the sphere with the combined (simd_quatf) DirectionalLight in the view, but the illumination (Z-axis) fails to properly illuminate the sphere with the expected result, compared to associated Meesus web page images. For the moment, I do not know how to correct the (Z-axis). Curious for a suggestion ... :] // Location values. let theLatitude: Double = 51.13107260 let theLongitude: Double = -114.01127910 let currentDate: Date = Date() struct TheCalculatedMoonPhaseTest_ContentView: View { var body: some View { VStack { if #available(macOS 15.0, *) { RealityView { content in let moonSphere_Entity = Entity.createSphere(radius: 0.90, color: .black) moonSphere.Entity.name = "MoonSphere" moonSphere.Entity.position = SIMD3<Float>(x: 0, y: 0, z: 0) content.add(moonSphere.Entity) let sunLight_Entity = createDirectionalLight(latitude: theLatitude, longitude: theLongitude, date: currentDate) content.add(sunLight_Entity) } // End of [RealityView] } else { // Earlier version required. } // End of [if #available(macOS 15.0, *)] } // End of [VStack] .background(Color.black) } // End of [var body: some View] // MARK: - 🟠🟠🟠🟠 [SET THE BACKGROUND COLOUR] 🟠🟠🟠🟠 var backgroundColor: Color = Color.init(.black) // MARK: - 🟠🟠🟠🟠 [CREATE THE DIRECTIONAL LIGHT FOR THE SPHERE] 🟠🟠🟠🟠 func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity { let directionalLight = DirectionalLight() directionalLight.light.color = .white directionalLight.light.intensity = 1000000 directionalLight.shadow = DirectionalLightComponent.Shadow() directionalLight.shadow?.maximumDistance = 5 directionalLight.shadow?.depthBias = 1 // MARK: 🟠🟠🟠🟠 Retrieve the [MEESUS MOON AGE VALUES] from the [CONSTANT FOLDER] 🟠🟠🟠🟠 let theMeesusMoonAge_LunarAgeDaysValue = 25.90567592898601 if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00) { let someCalculatedX_WestEastRadian: Float = Float(1.00) // Identify the sphere’s DirectionalLight Tilt Angle (Y) radian value :: // Note :: The following Tilt Angle is corrected to [Zenith] with the [MeesusCalculatedTilt_Angle] minus the [MeesusCalculatedPar_Angle]. let someCalculatedY_TiltAngleRadian: Float = Float(1.3396086) // Identify the sphere’s DirectionalLight Illumination (Z) radian Value :: // Note :: The Meesus calculated illumination fraction is converted to degrees, then converted to a radian value. let someCalculatedZ_IlluminationAngleRadian: Float = Float(0.45176168630244457) // <=== 14.3800% Illumination. // Define rotation angles in radians for X, Y, and Z axes. let x_Radians = someCalculatedX_WestEastRadian let y_Radians = someCalculatedY_TiltAngleRadian let z_Radians = someCalculatedZ_IlluminationAngleRadian // Identify and separate the quaternion [simd_quatf] for each Radian. let q_X = simd_quatf(angle: x_Radians, axis: SIMD3<Float>(1, 0, 0)) let q_Y = simd_quatf(angle: y_Radians, axis: SIMD3<Float>(0, 1, 0)) let q_Z = simd_quatf(angle: z_Radians, axis: SIMD3<Float>(0, 0, 1)) // Apply and combine the rotations, where order matters. let combinedRotation = q_Z * q_Y * q_X // Identify the [Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [combinedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians // Normalize the [combinedRotation]. let theNormalizesRotation = simd_normalize(combinedRotation) // Identify the [Normalized Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [normalizedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians // Assume the [theNormalizesRotation] appears reversed. let theCorrectedRotation = theNormalizesRotation.inverse // Identify the [Reversed Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [correctedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(-0.24427173, -0.61516714, 0.13599981)) ° Radians // Apply the [Corrected Rotation] to the entity. directionalLight.transform.rotation *= theCorrectedRotation // Add the [directionalLight] to the scene :: let anchor = AnchorEntity() anchor.addChild(directionalLight) } // End of [if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00)] return directionalLight } // End of [func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity] } // End of [struct TheCalculatedMoonPhaseTest_ContentView: View] // MARK: 🟠🟠🟠🟠 [ENTITY HELPER EXTENSION] 🟠🟠🟠🟠 extension Entity { static func createSphere(radius: Float, color: NSColor) -> Entity { let mesh = MeshResource.generateSphere(radius: radius) var material = PhysicallyBasedMaterial() material.baseColor = .init(tint: color) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) let entity = Entity() entity.components.set(modelComponent) entity.components.set(Transform()) return entity } // End of [static func createSphere(radius: Float, color: NSColor) -> Entity] } // End of [extension Entity] // Application Image :: Calgary // Website Image :: timeanddate // mooncalc.org
1
0
177
Feb ’26
Sticky Horizontal AnchorEntity
Hi, I'm trying to use AnchorEntity for horizontal surfaces. It works when the entity is being created, but I'm looking for a way to snap this entity to the nearest surface, after translating it, for example with a DragGesture. What would be the best way to achieve this? Using raycast, creating a new anchor, trackingMode to continuous etc. Do I need to use ARKitSession as I want continuous tracking?
1
0
278
Feb ’26
Nested RealityKit entity collisions priority
Hello, I'm struggling trying to interact with a RealityKit entity nested (or at least visually nested) in a second one. I think I've the same issue as mentioned on this StackOverflow post, but I can't manage to reproduce the solution. https://stackoverflow.com/questions/79244424/how-to-prioritize-a-specific-entity-when-collision-boxes-overlap-in-realitykit What I'd like to achieve is to translate the red box using a DragGesture, while still be able to interact using a TapGesture on the sphere. Currently, I can only do one at a time. Does anyone know the solution? extension CollisionGroup { static let parent: CollisionGroup = CollisionGroup(rawValue: 1 << 0) static let child: CollisionGroup = CollisionGroup(rawValue: 1 << 1) } struct ImmersiveView: View { var body: some View { RealityView { content in let boxMesh = MeshResource.generateBox(size: 0.35) let boxMaterial = SimpleMaterial(color: .red.withAlphaComponent(0.25), isMetallic: false) let boxEntity = ModelEntity(mesh: boxMesh, materials: [boxMaterial]) let sphereMesh = MeshResource.generateSphere(radius: 0.05) let sphereMaterial = SimpleMaterial(color: .blue, isMetallic: false) let sphereEntity = ModelEntity(mesh: sphereMesh, materials: [sphereMaterial]) content.add(sphereEntity) content.add(boxEntity) boxEntity.components.set(InputTargetComponent()) boxEntity.components.set( CollisionComponent( shapes: [ShapeResource.generateBox(size: SIMD3<Float>(repeating: 0.35))], isStatic: true, filter: CollisionFilter( group: .parent, mask: .parent.subtracting(.child) ) ) ) sphereEntity.components.set(InputTargetComponent()) sphereEntity.components.set(HoverEffectComponent()) sphereEntity.components.set( CollisionComponent( shapes: [ShapeResource.generateSphere(radius: 0.05)], isStatic: true, filter: CollisionFilter( group: .child, mask: .child.subtracting(.parent) ) ) ) } } }
3
0
309
Feb ’26
VideoMaterial Black Screen on Vision Pro Device (Works in Simulator)
VideoMaterial Black Screen on Vision Pro Device (Works in Simulator) App Overview App Name: Extn Browser Bundle ID: ai.extn.browser Purpose: A visionOS web browser that plays 360°/180° VR videos in an immersive sphere environment Development Environment & SDK Versions Component Version Xcode 26.2 Swift 6.2 visionOS Deployment Target 26.2 Swift Concurrency MainActor isolation enabled App is released in the TestFlight. Frameworks Used SwiftUI - UI framework RealityKit - 3D rendering, MeshResource, ModelEntity, VideoMaterial AVFoundation - AVPlayer, AVAudioSession WebKit - WKWebView for browser functionality Network - NWListener for local proxy server Sphere Video Mechanism The app creates an immersive 360° video experience using the following approach: // 1. Create sphere mesh (10 meter radius for immersive viewing) let mesh = MeshResource.generateSphere(radius: 10.0) // 2. Create initial transparent material var material = UnlitMaterial() material.color = .init(tint: .clear) // 3. Create entity and invert sphere (negative X scale) let sphere = ModelEntity(mesh: mesh, materials: [material]) sphere.scale = SIMD3<Float>(-1, 1, 1) // Inverts normals for inside-out viewing sphere.position = SIMD3<Float>(0, 1.5, 0) // Eye level // 4. Create AVPlayer with video URL let player = AVPlayer(url: videoURL) // 5. Configure audio session for visionOS let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory(.playback, mode: .moviePlayback, options: [.mixWithOthers]) try audioSession.setActive(true) // 6. Create VideoMaterial and apply to sphere let videoMaterial = VideoMaterial(avPlayer: player) if var modelComponent = sphere.components[ModelComponent.self] { modelComponent.materials = [videoMaterial] sphere.components.set(modelComponent) } // 7. Start playback player.play() ImmersiveSpace Configuration // browserApp.swift ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() .environment(appModel) } .immersionStyle(selection: .constant(.mixed), in: .mixed) Entitlements <!-- browser.entitlements --> <key>com.apple.security.app-sandbox</key> <true/> <key>com.apple.security.network.client</key> <true/> <key>com.apple.security.network.server</key> <true/> Info.plist Network Configuration <key>NSAppTransportSecurity</key> <dict> <key>NSAllowsArbitraryLoads</key> <true/> </dict> The Issue Behavior in Simulator: Video plays correctly on the inverted sphere surface - 360° video is visible and wraps around the user as expected. Behavior on Physical Vision Pro: The sphere displays a black screen. No video content is visible, though the sphere entity itself is present. Important: Not a DRM/Licensing Issue This issue is NOT related to Digital Rights Management (DRM) or FairPlay. I have tested with: Unlicensed raw MP4 video files (no DRM protection) Self-hosted video content with no copy protection Direct MP4 URLs from CDN without any licensing requirements The same black screen behavior occurs with all unprotected video sources, ruling out DRM as the cause. (Plain H.264 MP4, no DRM) Screen Recording: Working in Simulator The following screen recording demonstrates playing a 360° YouTube video in the immersive sphere on the visionOS Simulator: https://cdn.commenda.kr/screen-001.mov This confirms that the VideoMaterial and sphere rendering work correctly in the simulator, but the same setup shows a black screen on the physical Vision Pro device. Observations AVPlayer status reports .readyToPlay - The video appears to load successfully VideoMaterial is created without errors - No exceptions thrown Sphere entity renders - The geometry is visible (black surface) Audio session is configured - No errors during audio session setup Network requests succeed - The video URL is accessible from the device Same result with local/unprotected content - DRM is not a factor Console Logs (Device) The logging shows: Sphere created and added to scene AVPlayer created with correct URL VideoMaterial created and applied Player status transitions to .readyToPlay player.play() called successfully Rate shows 1.0 (playing) Despite all success indicators, the rendered output is black. Questions for Apple Are there known differences in VideoMaterial behavior between the visionOS Simulator and physical Vision Pro hardware? Does VideoMaterial(avPlayer:) require specific video codec/format requirements that differ on device? (The test video is a standard H.264 MP4) Is there a required Metal capability or GPU feature for VideoMaterial that may not be available in certain contexts on device? Does the immersion style (.mixed) affect VideoMaterial rendering on hardware? Are there additional entitlements required for video texture rendering in RealityKit on physical hardware? Attempted Solutions Configured AVAudioSession with .playback category Added delay before player.play() to ensure material is applied Verified sphere scale inversion (-1, 1, 1) Tested multiple video URLs (including raw, unlicensed MP4 files) Confirmed network connectivity on device Ruled out DRM/FairPlay issues by testing unprotected content Environment Details Device: Apple Vision Pro visionOS Version: 26.2 Xcode Version: 26.2 macOS Version: Darwin 25.2.0
0
0
264
Feb ’26
How to visualize AR-dependent app if not supported on Simulator for Swift Student Challenge?
Recently, applications for the Swift Student Challenge opened up. I noticed, that when selected "where to run your app" (mine was developed in Xcode 26), and you select Xcode26, there is a note underneath it that basically says all Xcode projects will be run on the simulator. What if my project is dependent on AR? How would I let the judges test my submission?
2
0
299
Feb ’26
Lighting disabled inside PortalComponent in Progressive/Full Immersion (visionOS 26.2)
Hello, I am experiencing an issue where lighting is disabled within a Portal Component specifically when using Progressive or Full Immersion styles in visionOS 26.2. [Steps to Reproduce] Create a scene in Reality Composer Pro with custom lighting (Directional Light, IBL, etc.) and load it as an Entity. Add a PortalComponent to this Entity. Observe the Entity in an ImmersiveSpace under different Immersion Styles. [Observed Behavior] Mixed Immersion: Lighting works as expected.Progressive / Full Immersion: Lighting is disabled, causing the content to render incorrectly. [Additional Context] Without PortalComponent: The same Reality Composer Pro scene renders lighting correctly across all Immersion Styles (Mixed, Progressive, and Full).Regression: In applications built with Xcode 16.4 / visionOS 2.5, lighting inside the Portal functioned correctly. This issue appears to have emerged with Xcode 26.2 and visionOS 26.2. [Questions] Is the disabling of lighting inside Portals during Full/Progressive Immersion an intended architectural change or optimization in visionOS 26.2? Are there any workarounds, such as specific PortalComponent configurations or new API flags, to re-enable lighting in these immersion modes? Thank you.
1
0
512
Feb ’26
ModelEntity position values are stale after world recenter (Crown long press)
Hi everyone, I’m working with RealityKit on visionOS and I’m seeing unexpected behavior when the user long-presses the Digital Crown, which recenters the world. Observed behavior: When the world is recentered via long-pressing the Crown, the models remain visually in the correct place (as expected). However, if I query the model’s position or transform immediately after recentering (e.g. entity.position or similar), I still get the old values from before recenter. As soon as I interact with the model using a gesture (drag/rotate/scale), the position updates and then querying it returns the correct, updated values. So effectively: Recenter happens Visual position is correct Programmatic position remains stale First gesture causes the position to “snap” to the correct updated value Questions: Is there any event, notification, or callback that fires when the world is recentered due to a long press of the Crown button? Is there a recommended way to get the updated world-space transform immediately after recenter, without waiting for a gesture? Is this expected behavior due to deferred/lazy transform updates in RealityKit? Right now it feels like recentering updates the coordinate system but doesn’t immediately commit new transform values to entities until some interaction occurs. Any guidance or best-practice patterns for handling this would be appreciated. Thanks!
2
0
683
Feb ’26
Showing 3D/AR content on multiple pages in iOS with RealityView
Hi. I'm trying to show 3D or AR content in multiple pages on an iOS app but I have found that if a RealityView is used earlier in a flow then future uses will never display the camera feed, even if those earlier uses do not use any spatial tracking. For example, in the following simplified view, the second page realityView will not display the camera feed even though the first view does not use it not start a tracking session. struct ContentView: View { var body: some View { NavigationStack { VStack { RealityView { content in content.camera = .virtual // showing model skipped for brevity } NavigationLink("Second Page") { RealityView { content in content.camera = .spatialTracking } } } } } } What is the way around this so that 3D content can be displayed in multiple places in the app without preventing AR content from working? I have also found the same problem when wrapping an ARView for use in SwiftUI.
0
0
313
Feb ’26
In SwiftUI on macOS, using instancing in RealityKit, how can I set individual colours per instance?
I have written this function: @available(macOS 26.0, *) func instancing() async -> Entity { let entity = Entity() do { // 1. Create a CustomMaterial let library = offscreenRenderer.pointRenderer!.device.makeDefaultLibrary()! let surfaceShader = CustomMaterial.SurfaceShader( named: "surfaceShaderWithCustomUniforms", // This must match the function name in Metal in: library ) let instanceCount = 10 // No idea how to actually use this... // let bufferSize = instanceCount * MemoryLayout<UInt32>.stride // // // Create the descriptor // var descriptor = LowLevelBuffer.Descriptor(capacity: bufferSize, sizeMultiple: MemoryLayout<UInt32>.stride) // // // Initialize the buffer // let lowLevelBuffer = try LowLevelBuffer(descriptor: descriptor) // lowLevelBuffer.withUnsafeMutableBytes { rawBytes in // // Bind the raw memory to the UInt32 type // let pointer = rawBytes.bindMemory(to: UInt32.self) // pointer[1] = 0xff_0000 // pointer[0] = 0x00_ff00 // pointer[2] = 0x00_00ff // pointer[3] = 0xff_ff00 // pointer[4] = 0xff_00ff // pointer[5] = 0x00_ffff // pointer[6] = 0xff_ffff // pointer[7] = 0x7f_0000 // pointer[8] = 0x00_7f00 // pointer[9] = 0x00_007f // } var material = try CustomMaterial(surfaceShader: surfaceShader, lightingModel: .lit) material.withMutableUniforms(ofType: SurfaceCustomUniforms.self, stage: .surfaceShader) { params, resources in params.argb = 0xff_0000 } // 2. Create the ModelComponent (provides the MESH and MATERIAL) let mesh = MeshResource.generateSphere(radius: 0.5) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) // 3. Create the MeshInstancesComponent (provides the INSTANCE TRANSFORMS) let instanceData = try LowLevelInstanceData(instanceCount: instanceCount) instanceData.withMutableTransforms { transforms in for i in 0..<instanceCount { let instanceAngle = 2 * .pi * Float(i) / Float(instanceCount) let radialTranslation: SIMD3<Float> = [-sin(instanceAngle), cos(instanceAngle), 0] * 4 // Position each sphere around a circle. let transform = Transform( scale: .one, rotation: simd_quatf(angle: instanceAngle, axis: [0, 0, 1]), translation: radialTranslation ) transforms[i] = transform.matrix } } let instancesComponent = try MeshInstancesComponent(mesh: mesh, instances: instanceData) // 4. Attach BOTH to the same entity entity.components.set(modelComponent) entity.components.set(instancesComponent) } catch { print("Failed to create mesh instances: \(error)") } return entity } and this is the corresponding Metal shader typedef struct { uint32_t argb; } SurfaceCustomUniforms; [[stitchable]] void surfaceShaderWithCustomUniforms(realitykit::surface_parameters params, constant SurfaceCustomUniforms &customParams) { half3 color = { static_cast<half>((customParams.argb >> 16) & 0xff), static_cast<half>((customParams.argb >> 8) & 0xff), static_cast<half>(customParams.argb & 0xff) }; params.surface().set_base_color(color); } which works well and generates 10 red spheres. While listening to the WWDC25 presentation on what's new in RealityKit I am positive to hear the presenter saying that it is possible to customise each instance using a LowLevelBuffer, but so far all my attempts have failed. Is it possible, and if so how ? Thanks for reading and for your help. Kind regards, Christian
1
1
262
Jan ’26
With manipulation component, once you let go, how to prevent the entity from disappearing while animating it back into the volume
So with the new ManipulationComponent, we can choose "stay" and then if you drag it out of your volume, once you let go it will instantly disappear. We can "animate" it back to inside the volume, eg.: content.subscribe(to: ManipulationEvents.WillRelease.self) { event in Entity.animate( .easeInOut(duration: 1), body: { event.entity.position = [0, 0.2, 0] }, completion: {} ) }, Howeve,r for the duration that it travels outside of the volume it's invisible the whole time. In this apple video, it seems to be visible when dragging and when letting go, but perhaps that's not a volume they're dragging it out of? https://youtu.be/VtenPKrvPOU?si=y1zoZOs2IMyDzOm6&t=1748 Does anyone know how to keep the entity visible even when after letting the entity go while you animate it back towards inside of your volume?
1
1
987
Jan ’26
The AccessoryAnchor transform does not match any of the Accessory.LocationName options.
I am using AccessoryTrackingProvider from ARKit to get the transform of the PSVR2 controller via originFromAnchorTransform of the AccessoryAnchor. I also am trying to use AnchorEntity on the controller using RealityKit However, none of the three options for Accessory.LocationName, which should be used to define the AnchorEntity target, seem to match the position on the controller which is being sent from ARKit. The picture attached is showing two transforms: RealityKit - using .gripSurface to define the AnchoringComponent.Target.accesssory location. ARKit - using originFromAnchorTransform for AccessoryTrackingProvider. They are not aligned at the same point. As for the other options of Accessory.LocationName, using .aim is located at the tip of the controller and .grip is the same position as .gripSurface but with a different orientation. I am wondering why there is not an option for Accessory.LocationName that actually matches the transform captured by ARKit?
3
0
1.5k
Jan ’26
Manipulation stops working when changing rooms
This post documents an issue I reported in feedback FB19610114 and see if anyone knows of a workaround. Here is a copy of the feedback. Short version Manipulation (SwiftUI OR RealityKit) fails to translate entities after changing rooms. By changing rooms, I mean a human wearing an Apple Vision Pro leaving one room and entering another room. Once this issue occurs, it impacts all apps that use these features. A device restart is the only solution I have to fix it. Feedback FB19610114 This is an odd one. I'm using the new Manipulation Component in visionOS 26. Most of the time this works well. Sometime it stops working and when it does the only way to get it working again is to reboot the headset. When this happens, I can continue to rotate and scale items, but translation no longer works. It is as if the item is stuck to a fixed point in the parent scene (window, volume, etc). When this bug occurs, it affects every app across the entire operating system that is using manipulation, including the RealityKit component AND the SwiftUI version. This is not limited to one app and is not limited to apps that I am working on. Once this error occurs, it affects literally any application across the operating system that is using this API, including apps from Apple. I won't speculate on the cause of this, but I do know of one way where I can always get it to happen. Here is how to reproduce it: Make an Xcode project with a single entity that uses the Manipulation Component. There is no need to customize the configuration of this component. The default implementation will work. Build and run this app on device. You can keep running from device or quit and launch the app like normal on device. Open the app and manipulate the entity - it should work as expected. Physically walk into another room. It is vital that you leave the current room that you are in and enter a different room entirely. Use the digital crown to recenter your view and bring your window or volume to you. Test the manipulation on the entity again - it should still be working as expected at this point. Physically, move yourself and your headset into the original room where you started. Use the digital crown to recenter your view and bring your window or volume to you. Test the manipulation on the entity again - you should now see the issue. When I follow the steps above, then 100% of the time manipulation translation stops working at this point. It will impact any application using this API. The only way to fix it is to restart my headset. A few points to keep in mind It does not matter if an app is actively being run from Xcode. When this occurs, it impacts every single app, not just one. When this occurs, rotation and scaling continue to work, but the entity/view cannot be translated. This impacts BOTH the SwiftUI version and the RealityKit version. When this occurs, the only way to "fix" it is to reboot the device.
6
3
993
Jan ’26
How to cast shadow on OcclusionMaterial in visionOS
I have a ModelEntity with GroundingShadowComponent entity.enumerateHierarchy { child, stop in child.components.set(GroundingShadowComponent(castsShadow: true)) } When I set it on the table, I can see the shadow on the table, even if I disable plane detection. However, when I enable plane detection, and the plane's material is OcclusionMaterial. I can not see the shadow on the table. As far as I know, receivesDynamicLighting is not usable in VisionOS. So how can I cast shadow on OcclusionMaterial in VisionOS? Or rather, is it possible to have the shadow properly displayed on the tabletop while ensuring that I cannot see objects beneath the table through it?
1
0
549
Jan ’26
Crash when Displaying RealityView on Multiple Screen only Connecting with Xcode
I have an iOS app that uses RealityView to display some models and interact with them, and the app uses regular iOS app navigations, then a challenge I'm facing is how to maintain multiple RealityView on multiplescreens. For example Screen A has a RealityView, and then I navigate to Screen B (also has a RealityView) using stack based navigation, when I do so I got a crash -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation Fragment Function(fsRealityPbr): argument envProbeTable[0] from Buffer(7) with offset(0) and length(16) has space for 16 bytes, but argument has a length(864). Fragment Function(fsRealityPbr): incorrect type of texture (MTLTextureType2D) bound at Texture binding at index 20 (expect MTLTextureTypeCubeArray) for envProbeDiffuseArray[0]. Interestingly this crash only happens when debugging with Xcode, not happens when the app runs on its own. I'm not sure what I'm doing is anti-pattern or it's some Xcode debugging limitation.
Replies
0
Boosts
0
Views
749
Activity
Mar ’26
Draw An Outline Around a Model Entity
Hi, Is there a resource or sample code about how to draw an outline around a mesh in RealityKit? Typically, this is useful for visualizing a selection, like in Reality Composer Pro. How to achieve such effect? A shader material? A post-process effect in ARView or RealityRenderer? Methods such as duplicating the entity mesh, scaling it, and using material.faceCulling = .front did not look good in my experiments. Thank you.
Replies
1
Boosts
0
Views
543
Activity
Feb ’26
Equivalent in RealityKit of SceneKit's SCNAction.customAction to run custom animation
I'm currently converting my game from SceneKit to RealityKit. What's the easiest way to run an animation in which every frame I want to run custom code, similar to SCNAction.customAction which accepts a callback that is called repeatedly until the action completes?
Replies
0
Boosts
0
Views
272
Activity
Feb ’26
Reality Kit Scene
Hi, I’m wondering whether RealityKit has its own scene management system, since it uses ARView (backed by ARKit) to present AR content. Does RealityKit manage scenes independently, or does it rely entirely on ARKit’s scene handling? Thank you.
Replies
1
Boosts
0
Views
156
Activity
Feb ’26
Unexpected behavior when writing entities and loading realityFiles.
I have a simple visionOS app that creates an Entity, writes it to the device, and then attempts to load it. However, when the entity file get overwritten, it affects the ability for the app to load it correctly. Here is my code for saving the entity. import SwiftUI import RealityKit import UniformTypeIdentifiers struct ContentView: View { var body: some View { VStack { ToggleImmersiveSpaceButton() Button("Save Entity") { Task { // if let entity = await buildEntityHierarchy(from: urdfPath) { let type = UTType.realityFile let filename = "testing.\(type.preferredFilenameExtension ?? "bin")" let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let fileURL = documentsURL.appendingPathComponent(filename) do { let mesh = MeshResource.generateBox(size: 1, cornerRadius: 0.05) let material = SimpleMaterial(color: .blue, isMetallic: true) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) let entity = Entity() entity.components.set(modelComponent) print("Writing \(fileURL)") try await entity.write(to: fileURL) } catch { print("Failed writing") } } } } .padding() } } Every time I press "Save Entity", I see a warning similar to: Writing file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality Failed to set dependencies on asset 1941054755064863441 because NetworkAssetManager does not have an asset entity for that id. When I open the immersive space, I attempt to load the same file: import SwiftUI import RealityKit import UniformTypeIdentifiers struct ImmersiveView: View { @Environment(AppModel.self) private var appModel var body: some View { RealityView { content in guard let type = UTType.realityFile.preferredFilenameExtension else { return } let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let fileURL = documentsURL.appendingPathComponent("testing.\(type)") guard FileManager.default.fileExists(atPath: fileURL.path) else { print("❌ File does not exist at path: \(fileURL.path)") return } if let entity = try? await Entity(contentsOf: fileURL) { content.add(entity) } } } } I also get errors after I overwrite the entity (by pressing "Save Entity" after I have successfully loaded it once). The warnings that appear when the Immersive space attempts to load the new entity are: Asset 13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'. Asset 8308977590385781534 Scene (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Scene_0.compiledscene failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to read archive entry. AssetLoadRequest failed because asset failed to load '13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh' (Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'.) The order of operations to make this happen: Launch app Press "Save Entity" to save the entity "Open Immersive Space" to view entity Press "Save Entity" to overwrite the entity "Open Immersive Space" to view entity, failed asset load request Also Launch app, the entity should still be save from last time the app ran "Open Immersive Space" to view entity Press "Save Entity" to overwrite the entity "Open Immersive Space" to view entity, failed asset load request NOTE: It appears I can get it to work slightly better by pressing the "Save Entity" button twice before attempting to view it again in the immersive space.
Replies
2
Boosts
3
Views
410
Activity
Feb ’26
RealityKit_DirectionalLight_Question
My application calculates three distinct Meesus Double [x, y, z] Radian values to light a sphere in RealityKit with DirectionalLight. It is my understanding that I must use (simd_quatf) for each radian value to properly light the sphere in the view. The code correctly [orientates] the sphere with the combined (simd_quatf) DirectionalLight in the view, but the illumination (Z-axis) fails to properly illuminate the sphere with the expected result, compared to associated Meesus web page images. For the moment, I do not know how to correct the (Z-axis). Curious for a suggestion ... :] // Location values. let theLatitude: Double = 51.13107260 let theLongitude: Double = -114.01127910 let currentDate: Date = Date() struct TheCalculatedMoonPhaseTest_ContentView: View { var body: some View { VStack { if #available(macOS 15.0, *) { RealityView { content in let moonSphere_Entity = Entity.createSphere(radius: 0.90, color: .black) moonSphere.Entity.name = "MoonSphere" moonSphere.Entity.position = SIMD3<Float>(x: 0, y: 0, z: 0) content.add(moonSphere.Entity) let sunLight_Entity = createDirectionalLight(latitude: theLatitude, longitude: theLongitude, date: currentDate) content.add(sunLight_Entity) } // End of [RealityView] } else { // Earlier version required. } // End of [if #available(macOS 15.0, *)] } // End of [VStack] .background(Color.black) } // End of [var body: some View] // MARK: - 🟠🟠🟠🟠 [SET THE BACKGROUND COLOUR] 🟠🟠🟠🟠 var backgroundColor: Color = Color.init(.black) // MARK: - 🟠🟠🟠🟠 [CREATE THE DIRECTIONAL LIGHT FOR THE SPHERE] 🟠🟠🟠🟠 func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity { let directionalLight = DirectionalLight() directionalLight.light.color = .white directionalLight.light.intensity = 1000000 directionalLight.shadow = DirectionalLightComponent.Shadow() directionalLight.shadow?.maximumDistance = 5 directionalLight.shadow?.depthBias = 1 // MARK: 🟠🟠🟠🟠 Retrieve the [MEESUS MOON AGE VALUES] from the [CONSTANT FOLDER] 🟠🟠🟠🟠 let theMeesusMoonAge_LunarAgeDaysValue = 25.90567592898601 if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00) { let someCalculatedX_WestEastRadian: Float = Float(1.00) // Identify the sphere’s DirectionalLight Tilt Angle (Y) radian value :: // Note :: The following Tilt Angle is corrected to [Zenith] with the [MeesusCalculatedTilt_Angle] minus the [MeesusCalculatedPar_Angle]. let someCalculatedY_TiltAngleRadian: Float = Float(1.3396086) // Identify the sphere’s DirectionalLight Illumination (Z) radian Value :: // Note :: The Meesus calculated illumination fraction is converted to degrees, then converted to a radian value. let someCalculatedZ_IlluminationAngleRadian: Float = Float(0.45176168630244457) // <=== 14.3800% Illumination. // Define rotation angles in radians for X, Y, and Z axes. let x_Radians = someCalculatedX_WestEastRadian let y_Radians = someCalculatedY_TiltAngleRadian let z_Radians = someCalculatedZ_IlluminationAngleRadian // Identify and separate the quaternion [simd_quatf] for each Radian. let q_X = simd_quatf(angle: x_Radians, axis: SIMD3<Float>(1, 0, 0)) let q_Y = simd_quatf(angle: y_Radians, axis: SIMD3<Float>(0, 1, 0)) let q_Z = simd_quatf(angle: z_Radians, axis: SIMD3<Float>(0, 0, 1)) // Apply and combine the rotations, where order matters. let combinedRotation = q_Z * q_Y * q_X // Identify the [Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [combinedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians // Normalize the [combinedRotation]. let theNormalizesRotation = simd_normalize(combinedRotation) // Identify the [Normalized Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [normalizedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians // Assume the [theNormalizesRotation] appears reversed. let theCorrectedRotation = theNormalizesRotation.inverse // Identify the [Reversed Combined Rotation]. // The [MyMoonMeesus] :: [WANING CRESCENT] calculated [correctedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(-0.24427173, -0.61516714, 0.13599981)) ° Radians // Apply the [Corrected Rotation] to the entity. directionalLight.transform.rotation *= theCorrectedRotation // Add the [directionalLight] to the scene :: let anchor = AnchorEntity() anchor.addChild(directionalLight) } // End of [if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00)] return directionalLight } // End of [func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity] } // End of [struct TheCalculatedMoonPhaseTest_ContentView: View] // MARK: 🟠🟠🟠🟠 [ENTITY HELPER EXTENSION] 🟠🟠🟠🟠 extension Entity { static func createSphere(radius: Float, color: NSColor) -> Entity { let mesh = MeshResource.generateSphere(radius: radius) var material = PhysicallyBasedMaterial() material.baseColor = .init(tint: color) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) let entity = Entity() entity.components.set(modelComponent) entity.components.set(Transform()) return entity } // End of [static func createSphere(radius: Float, color: NSColor) -> Entity] } // End of [extension Entity] // Application Image :: Calgary // Website Image :: timeanddate // mooncalc.org
Replies
1
Boosts
0
Views
177
Activity
Feb ’26
Sticky Horizontal AnchorEntity
Hi, I'm trying to use AnchorEntity for horizontal surfaces. It works when the entity is being created, but I'm looking for a way to snap this entity to the nearest surface, after translating it, for example with a DragGesture. What would be the best way to achieve this? Using raycast, creating a new anchor, trackingMode to continuous etc. Do I need to use ARKitSession as I want continuous tracking?
Replies
1
Boosts
0
Views
278
Activity
Feb ’26
Full Body Tracking
Hi, is there a way to track feet in a visionOS app in an immersive space? I want the whole body to be visible in VR, and I want to know if the user touches an object with their foot.
Replies
1
Boosts
0
Views
300
Activity
Feb ’26
Nested RealityKit entity collisions priority
Hello, I'm struggling trying to interact with a RealityKit entity nested (or at least visually nested) in a second one. I think I've the same issue as mentioned on this StackOverflow post, but I can't manage to reproduce the solution. https://stackoverflow.com/questions/79244424/how-to-prioritize-a-specific-entity-when-collision-boxes-overlap-in-realitykit What I'd like to achieve is to translate the red box using a DragGesture, while still be able to interact using a TapGesture on the sphere. Currently, I can only do one at a time. Does anyone know the solution? extension CollisionGroup { static let parent: CollisionGroup = CollisionGroup(rawValue: 1 << 0) static let child: CollisionGroup = CollisionGroup(rawValue: 1 << 1) } struct ImmersiveView: View { var body: some View { RealityView { content in let boxMesh = MeshResource.generateBox(size: 0.35) let boxMaterial = SimpleMaterial(color: .red.withAlphaComponent(0.25), isMetallic: false) let boxEntity = ModelEntity(mesh: boxMesh, materials: [boxMaterial]) let sphereMesh = MeshResource.generateSphere(radius: 0.05) let sphereMaterial = SimpleMaterial(color: .blue, isMetallic: false) let sphereEntity = ModelEntity(mesh: sphereMesh, materials: [sphereMaterial]) content.add(sphereEntity) content.add(boxEntity) boxEntity.components.set(InputTargetComponent()) boxEntity.components.set( CollisionComponent( shapes: [ShapeResource.generateBox(size: SIMD3<Float>(repeating: 0.35))], isStatic: true, filter: CollisionFilter( group: .parent, mask: .parent.subtracting(.child) ) ) ) sphereEntity.components.set(InputTargetComponent()) sphereEntity.components.set(HoverEffectComponent()) sphereEntity.components.set( CollisionComponent( shapes: [ShapeResource.generateSphere(radius: 0.05)], isStatic: true, filter: CollisionFilter( group: .child, mask: .child.subtracting(.parent) ) ) ) } } }
Replies
3
Boosts
0
Views
309
Activity
Feb ’26
VideoMaterial Black Screen on Vision Pro Device (Works in Simulator)
VideoMaterial Black Screen on Vision Pro Device (Works in Simulator) App Overview App Name: Extn Browser Bundle ID: ai.extn.browser Purpose: A visionOS web browser that plays 360°/180° VR videos in an immersive sphere environment Development Environment & SDK Versions Component Version Xcode 26.2 Swift 6.2 visionOS Deployment Target 26.2 Swift Concurrency MainActor isolation enabled App is released in the TestFlight. Frameworks Used SwiftUI - UI framework RealityKit - 3D rendering, MeshResource, ModelEntity, VideoMaterial AVFoundation - AVPlayer, AVAudioSession WebKit - WKWebView for browser functionality Network - NWListener for local proxy server Sphere Video Mechanism The app creates an immersive 360° video experience using the following approach: // 1. Create sphere mesh (10 meter radius for immersive viewing) let mesh = MeshResource.generateSphere(radius: 10.0) // 2. Create initial transparent material var material = UnlitMaterial() material.color = .init(tint: .clear) // 3. Create entity and invert sphere (negative X scale) let sphere = ModelEntity(mesh: mesh, materials: [material]) sphere.scale = SIMD3<Float>(-1, 1, 1) // Inverts normals for inside-out viewing sphere.position = SIMD3<Float>(0, 1.5, 0) // Eye level // 4. Create AVPlayer with video URL let player = AVPlayer(url: videoURL) // 5. Configure audio session for visionOS let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory(.playback, mode: .moviePlayback, options: [.mixWithOthers]) try audioSession.setActive(true) // 6. Create VideoMaterial and apply to sphere let videoMaterial = VideoMaterial(avPlayer: player) if var modelComponent = sphere.components[ModelComponent.self] { modelComponent.materials = [videoMaterial] sphere.components.set(modelComponent) } // 7. Start playback player.play() ImmersiveSpace Configuration // browserApp.swift ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() .environment(appModel) } .immersionStyle(selection: .constant(.mixed), in: .mixed) Entitlements <!-- browser.entitlements --> <key>com.apple.security.app-sandbox</key> <true/> <key>com.apple.security.network.client</key> <true/> <key>com.apple.security.network.server</key> <true/> Info.plist Network Configuration <key>NSAppTransportSecurity</key> <dict> <key>NSAllowsArbitraryLoads</key> <true/> </dict> The Issue Behavior in Simulator: Video plays correctly on the inverted sphere surface - 360° video is visible and wraps around the user as expected. Behavior on Physical Vision Pro: The sphere displays a black screen. No video content is visible, though the sphere entity itself is present. Important: Not a DRM/Licensing Issue This issue is NOT related to Digital Rights Management (DRM) or FairPlay. I have tested with: Unlicensed raw MP4 video files (no DRM protection) Self-hosted video content with no copy protection Direct MP4 URLs from CDN without any licensing requirements The same black screen behavior occurs with all unprotected video sources, ruling out DRM as the cause. (Plain H.264 MP4, no DRM) Screen Recording: Working in Simulator The following screen recording demonstrates playing a 360° YouTube video in the immersive sphere on the visionOS Simulator: https://cdn.commenda.kr/screen-001.mov This confirms that the VideoMaterial and sphere rendering work correctly in the simulator, but the same setup shows a black screen on the physical Vision Pro device. Observations AVPlayer status reports .readyToPlay - The video appears to load successfully VideoMaterial is created without errors - No exceptions thrown Sphere entity renders - The geometry is visible (black surface) Audio session is configured - No errors during audio session setup Network requests succeed - The video URL is accessible from the device Same result with local/unprotected content - DRM is not a factor Console Logs (Device) The logging shows: Sphere created and added to scene AVPlayer created with correct URL VideoMaterial created and applied Player status transitions to .readyToPlay player.play() called successfully Rate shows 1.0 (playing) Despite all success indicators, the rendered output is black. Questions for Apple Are there known differences in VideoMaterial behavior between the visionOS Simulator and physical Vision Pro hardware? Does VideoMaterial(avPlayer:) require specific video codec/format requirements that differ on device? (The test video is a standard H.264 MP4) Is there a required Metal capability or GPU feature for VideoMaterial that may not be available in certain contexts on device? Does the immersion style (.mixed) affect VideoMaterial rendering on hardware? Are there additional entitlements required for video texture rendering in RealityKit on physical hardware? Attempted Solutions Configured AVAudioSession with .playback category Added delay before player.play() to ensure material is applied Verified sphere scale inversion (-1, 1, 1) Tested multiple video URLs (including raw, unlicensed MP4 files) Confirmed network connectivity on device Ruled out DRM/FairPlay issues by testing unprotected content Environment Details Device: Apple Vision Pro visionOS Version: 26.2 Xcode Version: 26.2 macOS Version: Darwin 25.2.0
Replies
0
Boosts
0
Views
264
Activity
Feb ’26
How to visualize AR-dependent app if not supported on Simulator for Swift Student Challenge?
Recently, applications for the Swift Student Challenge opened up. I noticed, that when selected "where to run your app" (mine was developed in Xcode 26), and you select Xcode26, there is a note underneath it that basically says all Xcode projects will be run on the simulator. What if my project is dependent on AR? How would I let the judges test my submission?
Replies
2
Boosts
0
Views
299
Activity
Feb ’26
Lighting disabled inside PortalComponent in Progressive/Full Immersion (visionOS 26.2)
Hello, I am experiencing an issue where lighting is disabled within a Portal Component specifically when using Progressive or Full Immersion styles in visionOS 26.2. [Steps to Reproduce] Create a scene in Reality Composer Pro with custom lighting (Directional Light, IBL, etc.) and load it as an Entity. Add a PortalComponent to this Entity. Observe the Entity in an ImmersiveSpace under different Immersion Styles. [Observed Behavior] Mixed Immersion: Lighting works as expected.Progressive / Full Immersion: Lighting is disabled, causing the content to render incorrectly. [Additional Context] Without PortalComponent: The same Reality Composer Pro scene renders lighting correctly across all Immersion Styles (Mixed, Progressive, and Full).Regression: In applications built with Xcode 16.4 / visionOS 2.5, lighting inside the Portal functioned correctly. This issue appears to have emerged with Xcode 26.2 and visionOS 26.2. [Questions] Is the disabling of lighting inside Portals during Full/Progressive Immersion an intended architectural change or optimization in visionOS 26.2? Are there any workarounds, such as specific PortalComponent configurations or new API flags, to re-enable lighting in these immersion modes? Thank you.
Replies
1
Boosts
0
Views
512
Activity
Feb ’26
IKRig jitters with warning getMoCapTask
My procedural animation using IKRig occasionally jitters and throws this warning. It completely breaks immersion. getMoCapTask: Rig must be of type kCoreIKSolverTypePoseRetarget Any idea where to start looking or what could be causing this?
Replies
0
Boosts
0
Views
526
Activity
Feb ’26
ModelEntity position values are stale after world recenter (Crown long press)
Hi everyone, I’m working with RealityKit on visionOS and I’m seeing unexpected behavior when the user long-presses the Digital Crown, which recenters the world. Observed behavior: When the world is recentered via long-pressing the Crown, the models remain visually in the correct place (as expected). However, if I query the model’s position or transform immediately after recentering (e.g. entity.position or similar), I still get the old values from before recenter. As soon as I interact with the model using a gesture (drag/rotate/scale), the position updates and then querying it returns the correct, updated values. So effectively: Recenter happens Visual position is correct Programmatic position remains stale First gesture causes the position to “snap” to the correct updated value Questions: Is there any event, notification, or callback that fires when the world is recentered due to a long press of the Crown button? Is there a recommended way to get the updated world-space transform immediately after recenter, without waiting for a gesture? Is this expected behavior due to deferred/lazy transform updates in RealityKit? Right now it feels like recentering updates the coordinate system but doesn’t immediately commit new transform values to entities until some interaction occurs. Any guidance or best-practice patterns for handling this would be appreciated. Thanks!
Replies
2
Boosts
0
Views
683
Activity
Feb ’26
Showing 3D/AR content on multiple pages in iOS with RealityView
Hi. I'm trying to show 3D or AR content in multiple pages on an iOS app but I have found that if a RealityView is used earlier in a flow then future uses will never display the camera feed, even if those earlier uses do not use any spatial tracking. For example, in the following simplified view, the second page realityView will not display the camera feed even though the first view does not use it not start a tracking session. struct ContentView: View { var body: some View { NavigationStack { VStack { RealityView { content in content.camera = .virtual // showing model skipped for brevity } NavigationLink("Second Page") { RealityView { content in content.camera = .spatialTracking } } } } } } What is the way around this so that 3D content can be displayed in multiple places in the app without preventing AR content from working? I have also found the same problem when wrapping an ARView for use in SwiftUI.
Replies
0
Boosts
0
Views
313
Activity
Feb ’26
In SwiftUI on macOS, using instancing in RealityKit, how can I set individual colours per instance?
I have written this function: @available(macOS 26.0, *) func instancing() async -> Entity { let entity = Entity() do { // 1. Create a CustomMaterial let library = offscreenRenderer.pointRenderer!.device.makeDefaultLibrary()! let surfaceShader = CustomMaterial.SurfaceShader( named: "surfaceShaderWithCustomUniforms", // This must match the function name in Metal in: library ) let instanceCount = 10 // No idea how to actually use this... // let bufferSize = instanceCount * MemoryLayout<UInt32>.stride // // // Create the descriptor // var descriptor = LowLevelBuffer.Descriptor(capacity: bufferSize, sizeMultiple: MemoryLayout<UInt32>.stride) // // // Initialize the buffer // let lowLevelBuffer = try LowLevelBuffer(descriptor: descriptor) // lowLevelBuffer.withUnsafeMutableBytes { rawBytes in // // Bind the raw memory to the UInt32 type // let pointer = rawBytes.bindMemory(to: UInt32.self) // pointer[1] = 0xff_0000 // pointer[0] = 0x00_ff00 // pointer[2] = 0x00_00ff // pointer[3] = 0xff_ff00 // pointer[4] = 0xff_00ff // pointer[5] = 0x00_ffff // pointer[6] = 0xff_ffff // pointer[7] = 0x7f_0000 // pointer[8] = 0x00_7f00 // pointer[9] = 0x00_007f // } var material = try CustomMaterial(surfaceShader: surfaceShader, lightingModel: .lit) material.withMutableUniforms(ofType: SurfaceCustomUniforms.self, stage: .surfaceShader) { params, resources in params.argb = 0xff_0000 } // 2. Create the ModelComponent (provides the MESH and MATERIAL) let mesh = MeshResource.generateSphere(radius: 0.5) let modelComponent = ModelComponent(mesh: mesh, materials: [material]) // 3. Create the MeshInstancesComponent (provides the INSTANCE TRANSFORMS) let instanceData = try LowLevelInstanceData(instanceCount: instanceCount) instanceData.withMutableTransforms { transforms in for i in 0..<instanceCount { let instanceAngle = 2 * .pi * Float(i) / Float(instanceCount) let radialTranslation: SIMD3<Float> = [-sin(instanceAngle), cos(instanceAngle), 0] * 4 // Position each sphere around a circle. let transform = Transform( scale: .one, rotation: simd_quatf(angle: instanceAngle, axis: [0, 0, 1]), translation: radialTranslation ) transforms[i] = transform.matrix } } let instancesComponent = try MeshInstancesComponent(mesh: mesh, instances: instanceData) // 4. Attach BOTH to the same entity entity.components.set(modelComponent) entity.components.set(instancesComponent) } catch { print("Failed to create mesh instances: \(error)") } return entity } and this is the corresponding Metal shader typedef struct { uint32_t argb; } SurfaceCustomUniforms; [[stitchable]] void surfaceShaderWithCustomUniforms(realitykit::surface_parameters params, constant SurfaceCustomUniforms &customParams) { half3 color = { static_cast<half>((customParams.argb >> 16) & 0xff), static_cast<half>((customParams.argb >> 8) & 0xff), static_cast<half>(customParams.argb & 0xff) }; params.surface().set_base_color(color); } which works well and generates 10 red spheres. While listening to the WWDC25 presentation on what's new in RealityKit I am positive to hear the presenter saying that it is possible to customise each instance using a LowLevelBuffer, but so far all my attempts have failed. Is it possible, and if so how ? Thanks for reading and for your help. Kind regards, Christian
Replies
1
Boosts
1
Views
262
Activity
Jan ’26
With manipulation component, once you let go, how to prevent the entity from disappearing while animating it back into the volume
So with the new ManipulationComponent, we can choose "stay" and then if you drag it out of your volume, once you let go it will instantly disappear. We can "animate" it back to inside the volume, eg.: content.subscribe(to: ManipulationEvents.WillRelease.self) { event in Entity.animate( .easeInOut(duration: 1), body: { event.entity.position = [0, 0.2, 0] }, completion: {} ) }, Howeve,r for the duration that it travels outside of the volume it's invisible the whole time. In this apple video, it seems to be visible when dragging and when letting go, but perhaps that's not a volume they're dragging it out of? https://youtu.be/VtenPKrvPOU?si=y1zoZOs2IMyDzOm6&t=1748 Does anyone know how to keep the entity visible even when after letting the entity go while you animate it back towards inside of your volume?
Replies
1
Boosts
1
Views
987
Activity
Jan ’26
The AccessoryAnchor transform does not match any of the Accessory.LocationName options.
I am using AccessoryTrackingProvider from ARKit to get the transform of the PSVR2 controller via originFromAnchorTransform of the AccessoryAnchor. I also am trying to use AnchorEntity on the controller using RealityKit However, none of the three options for Accessory.LocationName, which should be used to define the AnchorEntity target, seem to match the position on the controller which is being sent from ARKit. The picture attached is showing two transforms: RealityKit - using .gripSurface to define the AnchoringComponent.Target.accesssory location. ARKit - using originFromAnchorTransform for AccessoryTrackingProvider. They are not aligned at the same point. As for the other options of Accessory.LocationName, using .aim is located at the tip of the controller and .grip is the same position as .gripSurface but with a different orientation. I am wondering why there is not an option for Accessory.LocationName that actually matches the transform captured by ARKit?
Replies
3
Boosts
0
Views
1.5k
Activity
Jan ’26
Manipulation stops working when changing rooms
This post documents an issue I reported in feedback FB19610114 and see if anyone knows of a workaround. Here is a copy of the feedback. Short version Manipulation (SwiftUI OR RealityKit) fails to translate entities after changing rooms. By changing rooms, I mean a human wearing an Apple Vision Pro leaving one room and entering another room. Once this issue occurs, it impacts all apps that use these features. A device restart is the only solution I have to fix it. Feedback FB19610114 This is an odd one. I'm using the new Manipulation Component in visionOS 26. Most of the time this works well. Sometime it stops working and when it does the only way to get it working again is to reboot the headset. When this happens, I can continue to rotate and scale items, but translation no longer works. It is as if the item is stuck to a fixed point in the parent scene (window, volume, etc). When this bug occurs, it affects every app across the entire operating system that is using manipulation, including the RealityKit component AND the SwiftUI version. This is not limited to one app and is not limited to apps that I am working on. Once this error occurs, it affects literally any application across the operating system that is using this API, including apps from Apple. I won't speculate on the cause of this, but I do know of one way where I can always get it to happen. Here is how to reproduce it: Make an Xcode project with a single entity that uses the Manipulation Component. There is no need to customize the configuration of this component. The default implementation will work. Build and run this app on device. You can keep running from device or quit and launch the app like normal on device. Open the app and manipulate the entity - it should work as expected. Physically walk into another room. It is vital that you leave the current room that you are in and enter a different room entirely. Use the digital crown to recenter your view and bring your window or volume to you. Test the manipulation on the entity again - it should still be working as expected at this point. Physically, move yourself and your headset into the original room where you started. Use the digital crown to recenter your view and bring your window or volume to you. Test the manipulation on the entity again - you should now see the issue. When I follow the steps above, then 100% of the time manipulation translation stops working at this point. It will impact any application using this API. The only way to fix it is to restart my headset. A few points to keep in mind It does not matter if an app is actively being run from Xcode. When this occurs, it impacts every single app, not just one. When this occurs, rotation and scaling continue to work, but the entity/view cannot be translated. This impacts BOTH the SwiftUI version and the RealityKit version. When this occurs, the only way to "fix" it is to reboot the device.
Replies
6
Boosts
3
Views
993
Activity
Jan ’26
How to cast shadow on OcclusionMaterial in visionOS
I have a ModelEntity with GroundingShadowComponent entity.enumerateHierarchy { child, stop in child.components.set(GroundingShadowComponent(castsShadow: true)) } When I set it on the table, I can see the shadow on the table, even if I disable plane detection. However, when I enable plane detection, and the plane's material is OcclusionMaterial. I can not see the shadow on the table. As far as I know, receivesDynamicLighting is not usable in VisionOS. So how can I cast shadow on OcclusionMaterial in VisionOS? Or rather, is it possible to have the shadow properly displayed on the tabletop while ensuring that I cannot see objects beneath the table through it?
Replies
1
Boosts
0
Views
549
Activity
Jan ’26