RealityKit

RSS for tag

Simulate and render 3D content for use in your augmented reality apps using RealityKit.

RealityKit Documentation

Posts under RealityKit subtopic

Post

Replies

Boosts

Views

Activity

Import models into my game
Hello, Im watching this tutorial: https://www.youtube.com/watch?v=tNkvu-YUVro&t=241s I have a reality composer project saved But she does not explain step by step what directory to import, how exactly to import it, and where exactly to import it to I tried to drag and drop every folder level into xcode and also into finder, 1 by 1, nothing works. it does not recognize my contentBundle in code. if someone from apple can upload 2 screenshots sowing the proper way to do this in 2026 it will be awesome Thanks guys :)
2
0
305
6d
How do I work with the BlendMask node in the animation graph in RCP3?
In RCP3, is it possible to mask out joints in a pose at runtime? I want to have two poses running for my character, one for the upper body and one for the lower body and I was hoping to do that from the animation graph. It seems like I should be able to have two State Machine nodes running, and use the Blend Mask node to make sure only joints from the upper body animation comes through from the upper body state machine, and the same for the lower body. The description of the Blend Mask node say "Filter a pose by applying per-joint weights from a blend mask" so that seems like what I want. But I can't figure out how to author a blend mask resources in RCP3. It seems to be possible to do programmatically via the SkeletonResource.BlendMask struct but how do I do it in RCP3? Also, the Blend Mask node takes two poses as parameters, whereas I was expecting it to take a pose and a blend mask.
0
0
113
1w
Why is this framework marketed as AR/VR ?
Its an honest question, while VR and AR are a cool thing, they are a bit far from what the vast majority of gaming is. (wither mobile or none mobile) Im only a few weeks with the system, but it think its insanely powerful as a game dev tool.Data orient based with ECS, fast, seamless, frictionless, well thought, optimized and packed everything needed. Just my humble opinion, Maybe consider naming it Apple Game Engine / Apple Game Framework - it will attract a lot of developers who just want to make games and are tired of the bloated and messy unreal and unity, tired of being chained into a bloated OOP hierarchy. "Reality" within the name of the kit, as a concept scares away a lot of developers, they think its just a tool only for VR, they dont know its actually a complete powerful and optimized framework to create any game you want. It have so many advantages over unreal. Unreal wraps everything in proprietary objects, forcing you learn mountains of made up keywords even if you already know native C++ A lot of developers are attracted to rayLib because of how bloated commercial game engines are, they dont know Reality Kit is actually a better solution then a lot of framework libraries out there. Just my thoughts,Maybe it finds its way to some folks in marketing. Either way thank you for creating this framework. its absolutely amazing to use
0
1
161
2w
Setting massProperties on a USDZ-loaded entity corrupts its transform (position/scale → NaN)
Setting massProperties on a USDZ-loaded entity corrupts its transform (position/scale → NaN) Category: RealityKit / visionOS Environment: visionOS 26.x Summary: Modifying PhysicsBodyComponent.massProperties at runtime on an entity loaded from a .usdz file silently corrupts the entity's transform. The translation and scale components become NaN, while rotation remains valid. The corruption occurs during the next RealityKit scene update cycle (e.g., during an await suspension on the main actor). Steps to Reproduce: Author a .usda file with a RigidBody component including valid m_mass and m_inertia values Load the entity at runtime via Entity(named: "MyEntity.usdz", in: bundle) Read the existing PhysicsBodyComponent, modify massProperties, and set it back: if var physics = entity.components[PhysicsBodyComponent.self] { physics.massProperties = .init( mass: 1.944, inertia: SIMD3<Float>(0.013, 0.026, 0.016), centerOfMass: (position: .zero, orientation: .init()) ) entity.components.set(physics) } Add the entity to the scene Perform any await call (e.g., TextureResource(contentsOf:)) that yields to the main actor, allowing a RealityKit scene update to run Expected: The entity retains its position and scale with the updated mass/inertia values. Actual: The entity's transform becomes corrupted: Transform( scale: SIMD3<Float>(nan, nan, nan), rotation: simd_quatf(real: 1.0, imag: SIMD3<Float>(0.0, 0.0, 0.0)), // ← fine translation: SIMD3<Float>(-nan, -nan, -nan) ) Additional findings from investigation: The entity's physics mode does not matter — corruption occurs even when the entity is .kinematic Creating a new PhysicsBodyComponent(...) and replacing the existing one also triggers the bug Modifying other properties on the same component (linearDamping, angularDamping, material, mode) does not cause corruption — only massProperties triggers it The parent entity's transform remains valid The computed mass/inertia values themselves are valid (finite, positive) The corruption is silent — no error, no warning, no crash from RealityKit itself Workaround: Author mass and inertia values directly in the .usda file and do not modify massProperties at runtime: def RealityKitStruct "massFrame" { float3 m_inertia = (0.02, 0.02, 0.038) float m_mass = 2.5 def RealityKitStruct "m_pose" { } } Other PhysicsBodyComponent properties (damping, material, mode) can safely be modified at runtime.
2
0
807
2w
Reality Kit 3 large scale practices - what is under the hood
Hello guys I want to use Reality Kit 3 for none AR games. Coming from unreal (7y) I have a few technical questions if i may please: (all questions are assuming my game will handle 50-500 enemies on screen) How does RTK3 handle everything under the hood? its easy to create a struct that hold multiple floats as a data blocks for the system to handle (like hp, dmg, ect), but, what happen when i want generic ASSET data (materials, meshes) ? What if i want to construct a material and hand a cheap id to the processor (system) ? while make sure i packed it nicely to the cpu's cache line . There is not much in the documentation to explain the under the hood architecture of the system for me to make educated decision abut my code. i try to avoid reference bloat in my game, and make sure the system does not coupled new material or new mesh every time it want to perform a generic command on the entity (in 60 fps) For example, in unreal mass, as bloated and over complicated it is - its pretty clear to me what is construction phase, what is the recommended way to generate my data blocks (called fragments there) and how exactly to use them in the system so its all tightly packed At this moment my only assumption is to create a 3 layers custom system to make sure its all running properly: The builder of the pool - it will do the " let shinyMetal = SimpleMaterial(color: ... ect" and save it on some dictionary map the data block - just a simple struct that hold a thin reference to the pool that can be use as a component in the ECS the system - The RTK3 built is system real time . can use the struct as a component and even swap the id in real time is this safe consider how RTK3 build under the hood? or is this even slower ? i have very little to go by. if reality kit engineer can reply and help me here it would be awesome :) Thank you so much for reading guys ! was kinda long. cheers
1
1
352
3w
MaterialX definitions download link is broken in ShaderGraph documentation
The link to download “MaterialX definitions” is broken on the main page of ShaderGraph documentation in the Interoperability section. See here: https://developer.apple.com/documentation/ShaderGraph#Interoperability Link in the page is https://developer.apple.com/augmented-reality/realitykit/files/MaterialX-definitions.zip but leads nowhere now. Since Reality Composer Pro 3 has been released, it would be good to have these definitions available to download again and to know if it’s still MaterialX 1.38 being used in the updated RCP & RealityKit. Opened a ticket about this: FB23258836
1
0
372
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Xcode hangs when I call PhotogrammetrySession(input: [PhotogrammetrySample]) with objectMask set and traps on some devices, see the attached screenshot, it gets to the function and hangs. Even the folder reconstruction also doesn't complete as it can't find alignment and displays the CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. In this case it failed while object masking was ON, so RealityKit could not find enough consistent feature tracks inside the masked pixels across the image set. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing. I will appreciate a timely response and willing to provide more clarity and informations, thank you so much for your understanding
4
0
2.0k
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample])
Please I would really appreciate a timely response to this: Providing a response and feedback to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing. Please I would really appreciate a timely response to this
1
0
479
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Providing clarity and response to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing.
0
0
949
May ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Providing a response and feedback to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 I have a created a feedback/bug report with ID FB22823733 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing.
0
0
1.5k
May ’26
RealityView AR - anchored to the screen not the floor
This started out as a plea for help, but in preparing this post I discovered the root cause. I'm posting it as a lesson learned in hopes it will help someone. I've spent a good chunk of March trying to get AR-mode working again in my unreleased game. I had it working with SceneKit and ARView 5 years ago, but since 2024 I've been converting the game to use RealityKit and RealityView on iOS, macOS, visionOS, and tvOS. I've been having no joy getting AR mode to work on iOS. I get the pass-through device video but the game content isn't anchored to the floor but rather anchored to the screen. I made a simple project with just a simple shape in the middle of a RealityView and an overlay with a SwiftUI toggle to go in and out of AR-mode. At first, my simple project worked, and I couldn't figure out what was different in the logic. Both projects used the same logic: func transitionToXR(_ content: inout RealityViewCameraContent) { content.remove(gameBoard.rootEntity) content.add(xrAnchor) content.camera = .spatialTracking Self.anchorStateChangedSubscription = content.subscribe(to: SceneEvents.AnchoredStateChanged.self) { event in if event.anchor == xrAnchor, event.isAnchored { xrAnchor.addChild(gameBoard.rootEntity) } } } Then I made an alternate version of my view, and reproduced the same "anchored to the screen not the floor" issue. I compared the code side-by-side and finally saw the difference! The one that didn't work, like my game, had a property 'cameraEntity' which is initialized with PerspectiveCamera(), position and look-at configured, then added as a child of the root entity. So, the simple fix was to remove 'cameraEntity' from the root entity before adding it to the detected AnchorEntity when going into AR-mode. Then when leaving AR-mode, I add back 'cameraEntity' as a child of the root entity and configure it again. So the lesson learned is: make sure there isn't a PerspectiveCamera in the tree of Entities added to an AnchorEntity with a .spatialTracking content camera. Apple: let me know if you think this is a bug or if I was being dumb. If a bug, I can use Feedback Assistant to report this. If I was being dumb, it wouldn't be the first time. :-)
5
0
698
May ’26
RealityKit custom component: `has()` returns `true` but typed subscript returns `nil` in SwiftPM test runner
swift test (SwiftPM CLI) fails to decode RealityKit custom components from USD files, even though entity.components.has(MyComponent.self) returns true. Typed access via entity.components[MyComponent.self] returns nil. This forces projects that use RealityKit custom components to use xcodebuild test exclusively. Minimal repro: github.com/mesqueeb/swiftpm-realitykit-custom-component-repro Repro steps git clone https://github.com/mesqueeb/swiftpm-realitykit-custom-component-repro cd swiftpm-realitykit-custom-component-repro swift test --filter componentsPresentButNotDecodableInSwiftTest Observed ✅ entity.components.has(ReproComponent.self) returns true ❌ entity.components[ReproComponent.self] returns nil Expected If has(...) returns true for a registered custom component, typed lookup should decode and return non-nil. Notes Running the same test via xcodebuild test works correctly The component is properly registered and the USDA file correctly references it This affects any project that relies on custom RealityKit components in tests — there is no swift test workaround Feedback ID: FB22099519 Environment: macOS 15.5, Xcode 16.4, Swift 6.1
1
0
859
May ’26
RealityKit crashes when rendering SpriteKit scene with SKShapeNode in postProcess callback
I'm converting my game from SceneKit to RealityKit. It has a SpriteKit overlay that according to Explore advanced rendering with RealityKit 2 I can add with the code below. The code runs fine if the SKScene only contains a SKSpriteNode (see the commented out line), but when I add a SKShapeNode with a fillColor instead, the app crashes with this error: -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation MTLDepthStencilDescriptor uses frontFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture MTLDepthStencilDescriptor uses backFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture ' I don't know enough about low-level graphics and stencils yet to figure out a quick solution, so I would appreciate if anyone could share an easy fix or explanation of what's wrong. Thanks! class ViewController: NSViewController { var device: MTLDevice! var renderer: SKRenderer! override func loadView() { let arView = ARView(frame: NSScreen.main!.frame) view = arView arView.renderCallbacks.prepareWithDevice = { [weak self] device in guard let self = self else { return } self.device = device renderer = SKRenderer(device: MTLCreateSystemDefaultDevice()!) let scene = SKScene() let shape = SKShapeNode(rectOf: CGSize(width: 10, height: 10)) shape.fillColor = .red scene.addChild(shape) // scene.addChild(SKSpriteNode(color: .red, size: CGSize(width: 10, height: 10))) renderer.scene = scene } arView.renderCallbacks.postProcess = { [weak self] context in guard let self = self else { return } let encoder = context.commandBuffer.makeBlitCommandEncoder() encoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture) encoder?.endEncoding() renderer.update(atTime: context.time) let descriptor = MTLRenderPassDescriptor() descriptor.colorAttachments[0].loadAction = .load descriptor.colorAttachments[0].storeAction = .store descriptor.colorAttachments[0].texture = context.targetColorTexture renderer.render(withViewport: CGRect(x: 0, y: 0, width: context.targetColorTexture.width, height: context.targetColorTexture.height), commandBuffer: context.commandBuffer, renderPassDescriptor: descriptor) } } }
9
0
2.9k
Apr ’26
ParticleEmitterComponent Position Offset Issue After iOS 26.1 Update – Seeking Solutions & Workarounds
Problem Summary After upgrading to iOS 26.1 and 26.2, I'm experiencing a particle positioning bug in RealityKit where ParticleEmitterComponent particles render at an incorrect offset relative to their parent entity. This behavior does not occur on iOS 18.6.2 or earlier versions, suggesting a regression introduced in the newer OS builds. Environment Details Operating System: iOS 26.1 & iOS 26.2 Framework: RealityKit Xcode Version: 16.2 (16C5032a) Expected vs. Actual Behavior Expected: Particles should render at the position of the entity to which the ParticleEmitterComponent is attached, matching the behavior on iOS 18.6.2 and earlier. Actual: Particles appear away from their parent entity, creating a visual misalignment that breaks the intended AR experience. Steps to Reproduce Create or open an AR application with RealityKit that uses particle components Attach a ParticleEmitterComponent to an entity via a custom system Run the application on iOS 26.1 or iOS 26.2 Observe that particles render at an offset position away from the entity Minimal Code Example Here's the setup from my test case: Custom Component & System: struct SparkleComponent4: Component {} class SparkleSystem4: System { static let query = EntityQuery(where: .has(SparkleComponent4.self)) required init(scene: Scene) {} func update(context: SceneUpdateContext) { for entity in context.scene.performQuery(Self.query) { // Only add once if entity.components.has(ParticleEmitterComponent.self) { continue } var newEmitter = ParticleEmitterComponent() newEmitter.mainEmitter.color = .constant(.single(.red)) entity.components.set(newEmitter) } } } AR Setup: let material = SimpleMaterial(color: .gray, roughness: 0.15, isMetallic: true) let model = Entity() model.components.set(ModelComponent(mesh: boxMesh, materials: [material])) model.components.set(SparkleComponent4()) model.position = [0, 0.05, 0] model.name = "MyCube" let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: [0.2, 0.2])) anchor.addChild(model) arView.scene.addAnchor(anchor) Questions for the Community Has anyone else encountered this particle positioning issue after updating to iOS 26.1/26.2? Are there known workarounds or configuration changes to ParticleEmitterComponent that restore correct positioning? Is this a confirmed bug, or could there be a change in coordinate system handling or transform inheritance that I'm missing? Additional Information I've already submitted this issue via Feedback Assistant(FB21346746)
3
1
1.6k
Apr ’26
RealityKit fill the background environment
I am new to RealityKit and Metal and I am building a RealityKit app that renders a procedural LowLevelMesh road. But the left and right side of the road is a complete green terrain mesh object and it doesn't look great. What I want is to add some rocks, tall trees and dence bushes (or weed) to make it look like the player is in the woods. But when I add many of those objects then the performance drains. What is the best approach to fill background empty spaces in the scene?
3
0
874
Apr ’26
Blending walk and run animations in RealityKit
Hi everybody, I have 2 separate animations run.usdz and walk.usdz animation files which are loaded perfectly in Reality Composer Pro and in the RealityKit application. I want to gradually increase the speed of my player by switching blending weight values from 0.0 (walking) to 1.0 (full speed running). let rabbit = await RabbitBuilder.loadWalkingRabbit() let runningRabbit = await RabbitBuilder.loadRunningRabbit() rabbit.scale = SIMD3(0.05, 0.05, 0.05) runningRabbit.scale = SIMD3(0.05, 0.05, 0.05) let walkAnimation = rabbit.availableAnimations let runAnimation = runningRabbit.availableAnimations RabbitWalker.walkAnim = walkAnimation.first! RabbitWalker.runAnim = runAnimation.first! guard let walk = RabbitWalker.walkAnim, let run = RabbitWalker.runAnim else { return } let blendTree = BlendTreeAnimation<JointTransforms>( BlendTreeBlendNode(sources: [ BlendTreeSourceNode(source: walk.definition, name: "walk", weight: .value(1 - weight)), BlendTreeSourceNode(source: run.definition, name: "run", weight: .value(weight)) ]), name: "rabbitLocomotion", repeatMode: .repeat, offset: TimeInterval(elapsed) ) // I have runtime error after executing this line: "Cannot add incompatible timeline type to blend tree." guard let resource = try? AnimationResource.generate(with: blendTree) else { return } entity.playAnimation(resource) static func loadWalkingRabbit() async -> Entity? { do { let scene = try await Entity(named: "Scene", in: realityKitEnvironmentBundle) guard let rabbit = await scene.findEntity(named: "RabbitWalk") else { return nil } await rabbit.removeFromParent() return rabbit } catch { return nil } } static func loadRunningRabbit() async -> Entity? { do { let scene = try await Entity(named: "Scene", in: realityKitEnvironmentBundle) guard let rabbit = await scene.findEntity(named: "RabbitRun") else { return nil } await rabbit.removeFromParent() return rabbit } catch { return nil } } But when I run this code I have this error; Cannot add incompatible timeline type to blend tree. By the way I have looked to developer's sample codes from here but I couldn't find any relevant BlendTreeAnimation sample which blends 2 animations. I would very happy if someone could direct me to a solution. Regards.
4
0
791
Apr ’26
Gestures not working correctly when setting the fov orientation to .horizontal
Hi there, I've discovered an issue with gesture handling in RealityKit when setting the camera’s fieldOfViewOrientation to horizontal. For instance, if I render a simple box at the center of the view with a collision shape that exactly matches its dimensions, the actual hit area behaves as if it's smaller than the box. Additionally, when attempting to drag the box away from the center, the hit area appears misaligned—offset slightly towards the center. Since the default fieldOfViewOrientation is vertical and everything works as expected in that mode, it seems that the gesture system might be assuming a vertical FOV. Given that the API allows setting it to horizontal, perhaps gestures should function correctly regardless of the orientation? Thank you!
2
0
1.2k
Apr ’26
Does VisionOS have the equivalent of ARView.physicsOrigin?
I'm trying to scale the physics of a scene without changing the apparent size to avoid the low-speed zeroing-out of motion that the physics simulation does. I found a technique for using separate simulation and physics roots in the docs, but it relies on ARView, which VisionOS doesn't have. This seems more elegant than scaling absolutely everything with shared root -- any chance I'm just failing in my searches to find the equivalent functionality?
3
1
1.1k
Mar ’26
RealityKit .Kinematic + collisions (visionOs)
Hi everyone, I'm new to visionOS development. I'm trying to create a physics-based scene (with gravity) where users can pick up and move objects on a workbench. I am struggling with physics interactions during the drag gesture: Kinematic Mode: If I switch to .kinematic during the drag, the object moves smoothly but clips through other objects (no collisions). Dynamic Mode: I tried keeping it .dynamic and applying linear velocity toward the hand position, but the movement feels laggy and unresponsive. Hybrid Approach: I tried switching to .kinematic during DragGesture.onChange and back to .dynamic on collision, but this causes the entity to jitter/shake violently when touching other objects. Has anyone found a clean way to drag objects while maintaining solid collisions. Thanks for your help!
3
0
1.2k
Mar ’26
RealityKit equivalent of ARGeoAnchor?
In ARKit there is ARGeoAnchor, which lets you anchor content using latitude and longitude so objects stay fixed to a real-world location. Is there an equivalent feature in RealityKit? I want to place points in the world and make sure they don't move or drift after placement. If RealityKit doesn't support this directly, what is the recommended approach?
0
1
864
Mar ’26
Import models into my game
Hello, Im watching this tutorial: https://www.youtube.com/watch?v=tNkvu-YUVro&t=241s I have a reality composer project saved But she does not explain step by step what directory to import, how exactly to import it, and where exactly to import it to I tried to drag and drop every folder level into xcode and also into finder, 1 by 1, nothing works. it does not recognize my contentBundle in code. if someone from apple can upload 2 screenshots sowing the proper way to do this in 2026 it will be awesome Thanks guys :)
Replies
2
Boosts
0
Views
305
Activity
6d
How do I work with the BlendMask node in the animation graph in RCP3?
In RCP3, is it possible to mask out joints in a pose at runtime? I want to have two poses running for my character, one for the upper body and one for the lower body and I was hoping to do that from the animation graph. It seems like I should be able to have two State Machine nodes running, and use the Blend Mask node to make sure only joints from the upper body animation comes through from the upper body state machine, and the same for the lower body. The description of the Blend Mask node say "Filter a pose by applying per-joint weights from a blend mask" so that seems like what I want. But I can't figure out how to author a blend mask resources in RCP3. It seems to be possible to do programmatically via the SkeletonResource.BlendMask struct but how do I do it in RCP3? Also, the Blend Mask node takes two poses as parameters, whereas I was expecting it to take a pose and a blend mask.
Replies
0
Boosts
0
Views
113
Activity
1w
Why is this framework marketed as AR/VR ?
Its an honest question, while VR and AR are a cool thing, they are a bit far from what the vast majority of gaming is. (wither mobile or none mobile) Im only a few weeks with the system, but it think its insanely powerful as a game dev tool.Data orient based with ECS, fast, seamless, frictionless, well thought, optimized and packed everything needed. Just my humble opinion, Maybe consider naming it Apple Game Engine / Apple Game Framework - it will attract a lot of developers who just want to make games and are tired of the bloated and messy unreal and unity, tired of being chained into a bloated OOP hierarchy. "Reality" within the name of the kit, as a concept scares away a lot of developers, they think its just a tool only for VR, they dont know its actually a complete powerful and optimized framework to create any game you want. It have so many advantages over unreal. Unreal wraps everything in proprietary objects, forcing you learn mountains of made up keywords even if you already know native C++ A lot of developers are attracted to rayLib because of how bloated commercial game engines are, they dont know Reality Kit is actually a better solution then a lot of framework libraries out there. Just my thoughts,Maybe it finds its way to some folks in marketing. Either way thank you for creating this framework. its absolutely amazing to use
Replies
0
Boosts
1
Views
161
Activity
2w
Setting massProperties on a USDZ-loaded entity corrupts its transform (position/scale → NaN)
Setting massProperties on a USDZ-loaded entity corrupts its transform (position/scale → NaN) Category: RealityKit / visionOS Environment: visionOS 26.x Summary: Modifying PhysicsBodyComponent.massProperties at runtime on an entity loaded from a .usdz file silently corrupts the entity's transform. The translation and scale components become NaN, while rotation remains valid. The corruption occurs during the next RealityKit scene update cycle (e.g., during an await suspension on the main actor). Steps to Reproduce: Author a .usda file with a RigidBody component including valid m_mass and m_inertia values Load the entity at runtime via Entity(named: "MyEntity.usdz", in: bundle) Read the existing PhysicsBodyComponent, modify massProperties, and set it back: if var physics = entity.components[PhysicsBodyComponent.self] { physics.massProperties = .init( mass: 1.944, inertia: SIMD3<Float>(0.013, 0.026, 0.016), centerOfMass: (position: .zero, orientation: .init()) ) entity.components.set(physics) } Add the entity to the scene Perform any await call (e.g., TextureResource(contentsOf:)) that yields to the main actor, allowing a RealityKit scene update to run Expected: The entity retains its position and scale with the updated mass/inertia values. Actual: The entity's transform becomes corrupted: Transform( scale: SIMD3<Float>(nan, nan, nan), rotation: simd_quatf(real: 1.0, imag: SIMD3<Float>(0.0, 0.0, 0.0)), // ← fine translation: SIMD3<Float>(-nan, -nan, -nan) ) Additional findings from investigation: The entity's physics mode does not matter — corruption occurs even when the entity is .kinematic Creating a new PhysicsBodyComponent(...) and replacing the existing one also triggers the bug Modifying other properties on the same component (linearDamping, angularDamping, material, mode) does not cause corruption — only massProperties triggers it The parent entity's transform remains valid The computed mass/inertia values themselves are valid (finite, positive) The corruption is silent — no error, no warning, no crash from RealityKit itself Workaround: Author mass and inertia values directly in the .usda file and do not modify massProperties at runtime: def RealityKitStruct "massFrame" { float3 m_inertia = (0.02, 0.02, 0.038) float m_mass = 2.5 def RealityKitStruct "m_pose" { } } Other PhysicsBodyComponent properties (damping, material, mode) can safely be modified at runtime.
Replies
2
Boosts
0
Views
807
Activity
2w
Reality Kit 3 large scale practices - what is under the hood
Hello guys I want to use Reality Kit 3 for none AR games. Coming from unreal (7y) I have a few technical questions if i may please: (all questions are assuming my game will handle 50-500 enemies on screen) How does RTK3 handle everything under the hood? its easy to create a struct that hold multiple floats as a data blocks for the system to handle (like hp, dmg, ect), but, what happen when i want generic ASSET data (materials, meshes) ? What if i want to construct a material and hand a cheap id to the processor (system) ? while make sure i packed it nicely to the cpu's cache line . There is not much in the documentation to explain the under the hood architecture of the system for me to make educated decision abut my code. i try to avoid reference bloat in my game, and make sure the system does not coupled new material or new mesh every time it want to perform a generic command on the entity (in 60 fps) For example, in unreal mass, as bloated and over complicated it is - its pretty clear to me what is construction phase, what is the recommended way to generate my data blocks (called fragments there) and how exactly to use them in the system so its all tightly packed At this moment my only assumption is to create a 3 layers custom system to make sure its all running properly: The builder of the pool - it will do the " let shinyMetal = SimpleMaterial(color: ... ect" and save it on some dictionary map the data block - just a simple struct that hold a thin reference to the pool that can be use as a component in the ECS the system - The RTK3 built is system real time . can use the struct as a component and even swap the id in real time is this safe consider how RTK3 build under the hood? or is this even slower ? i have very little to go by. if reality kit engineer can reply and help me here it would be awesome :) Thank you so much for reading guys ! was kinda long. cheers
Replies
1
Boosts
1
Views
352
Activity
3w
MaterialX definitions download link is broken in ShaderGraph documentation
The link to download “MaterialX definitions” is broken on the main page of ShaderGraph documentation in the Interoperability section. See here: https://developer.apple.com/documentation/ShaderGraph#Interoperability Link in the page is https://developer.apple.com/augmented-reality/realitykit/files/MaterialX-definitions.zip but leads nowhere now. Since Reality Composer Pro 3 has been released, it would be good to have these definitions available to download again and to know if it’s still MaterialX 1.38 being used in the updated RCP & RealityKit. Opened a ticket about this: FB23258836
Replies
1
Boosts
0
Views
372
Activity
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Xcode hangs when I call PhotogrammetrySession(input: [PhotogrammetrySample]) with objectMask set and traps on some devices, see the attached screenshot, it gets to the function and hangs. Even the folder reconstruction also doesn't complete as it can't find alignment and displays the CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. In this case it failed while object masking was ON, so RealityKit could not find enough consistent feature tracks inside the masked pixels across the image set. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing. I will appreciate a timely response and willing to provide more clarity and informations, thank you so much for your understanding
Replies
4
Boosts
0
Views
2.0k
Activity
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample])
Please I would really appreciate a timely response to this: Providing a response and feedback to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing. Please I would really appreciate a timely response to this
Replies
1
Boosts
0
Views
479
Activity
Jun ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Providing clarity and response to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing.
Replies
0
Boosts
0
Views
949
Activity
May ’26
PhotogrammetrySession(input: [PhotogrammetrySample]) Hangs or terminates
Providing a response and feedback to this: https://origin-devforums.apple.com/forums/thread/827311 and https://developer.apple.com/forums/thread/827043?page=1#889020022 I have a created a feedback/bug report with ID FB22823733 Feedback Report: I have created a feedback report as well like recommended with this ID: FB22823733, with more elaborate images of my implementation, also see here More clarity: Yes, this is for an iOS app(iOS 18+) but testing and debugging on Xcode(MacOS 15.7.7, M4 24GB), which traps or hangs on getting to [PhotogrammetrySample] even while using the lazy sequence and the contentsOf as specified in your docs. I also tried using the PhotogrammetrySession folder run but its still failed with : CoreOC.PhotogrammetrySession.Error 6 and I understand to mean alignment failed. What can be done or do you guys expose any functions that can be used to aid, or handle these internally, can't find any internally. The ObjectMasks are actually segmentation masks from an ML algorithm . To replicate try calling PhotogrammetrySession(input: [PhotogrammetrySample]) with contentsOf as captured on your documentation, even with like 30 image set or is there something I'm missing.
Replies
0
Boosts
0
Views
1.5k
Activity
May ’26
RealityView AR - anchored to the screen not the floor
This started out as a plea for help, but in preparing this post I discovered the root cause. I'm posting it as a lesson learned in hopes it will help someone. I've spent a good chunk of March trying to get AR-mode working again in my unreleased game. I had it working with SceneKit and ARView 5 years ago, but since 2024 I've been converting the game to use RealityKit and RealityView on iOS, macOS, visionOS, and tvOS. I've been having no joy getting AR mode to work on iOS. I get the pass-through device video but the game content isn't anchored to the floor but rather anchored to the screen. I made a simple project with just a simple shape in the middle of a RealityView and an overlay with a SwiftUI toggle to go in and out of AR-mode. At first, my simple project worked, and I couldn't figure out what was different in the logic. Both projects used the same logic: func transitionToXR(_ content: inout RealityViewCameraContent) { content.remove(gameBoard.rootEntity) content.add(xrAnchor) content.camera = .spatialTracking Self.anchorStateChangedSubscription = content.subscribe(to: SceneEvents.AnchoredStateChanged.self) { event in if event.anchor == xrAnchor, event.isAnchored { xrAnchor.addChild(gameBoard.rootEntity) } } } Then I made an alternate version of my view, and reproduced the same "anchored to the screen not the floor" issue. I compared the code side-by-side and finally saw the difference! The one that didn't work, like my game, had a property 'cameraEntity' which is initialized with PerspectiveCamera(), position and look-at configured, then added as a child of the root entity. So, the simple fix was to remove 'cameraEntity' from the root entity before adding it to the detected AnchorEntity when going into AR-mode. Then when leaving AR-mode, I add back 'cameraEntity' as a child of the root entity and configure it again. So the lesson learned is: make sure there isn't a PerspectiveCamera in the tree of Entities added to an AnchorEntity with a .spatialTracking content camera. Apple: let me know if you think this is a bug or if I was being dumb. If a bug, I can use Feedback Assistant to report this. If I was being dumb, it wouldn't be the first time. :-)
Replies
5
Boosts
0
Views
698
Activity
May ’26
RealityKit custom component: `has()` returns `true` but typed subscript returns `nil` in SwiftPM test runner
swift test (SwiftPM CLI) fails to decode RealityKit custom components from USD files, even though entity.components.has(MyComponent.self) returns true. Typed access via entity.components[MyComponent.self] returns nil. This forces projects that use RealityKit custom components to use xcodebuild test exclusively. Minimal repro: github.com/mesqueeb/swiftpm-realitykit-custom-component-repro Repro steps git clone https://github.com/mesqueeb/swiftpm-realitykit-custom-component-repro cd swiftpm-realitykit-custom-component-repro swift test --filter componentsPresentButNotDecodableInSwiftTest Observed ✅ entity.components.has(ReproComponent.self) returns true ❌ entity.components[ReproComponent.self] returns nil Expected If has(...) returns true for a registered custom component, typed lookup should decode and return non-nil. Notes Running the same test via xcodebuild test works correctly The component is properly registered and the USDA file correctly references it This affects any project that relies on custom RealityKit components in tests — there is no swift test workaround Feedback ID: FB22099519 Environment: macOS 15.5, Xcode 16.4, Swift 6.1
Replies
1
Boosts
0
Views
859
Activity
May ’26
RealityKit crashes when rendering SpriteKit scene with SKShapeNode in postProcess callback
I'm converting my game from SceneKit to RealityKit. It has a SpriteKit overlay that according to Explore advanced rendering with RealityKit 2 I can add with the code below. The code runs fine if the SKScene only contains a SKSpriteNode (see the commented out line), but when I add a SKShapeNode with a fillColor instead, the app crashes with this error: -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation MTLDepthStencilDescriptor uses frontFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture MTLDepthStencilDescriptor uses backFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture ' I don't know enough about low-level graphics and stencils yet to figure out a quick solution, so I would appreciate if anyone could share an easy fix or explanation of what's wrong. Thanks! class ViewController: NSViewController { var device: MTLDevice! var renderer: SKRenderer! override func loadView() { let arView = ARView(frame: NSScreen.main!.frame) view = arView arView.renderCallbacks.prepareWithDevice = { [weak self] device in guard let self = self else { return } self.device = device renderer = SKRenderer(device: MTLCreateSystemDefaultDevice()!) let scene = SKScene() let shape = SKShapeNode(rectOf: CGSize(width: 10, height: 10)) shape.fillColor = .red scene.addChild(shape) // scene.addChild(SKSpriteNode(color: .red, size: CGSize(width: 10, height: 10))) renderer.scene = scene } arView.renderCallbacks.postProcess = { [weak self] context in guard let self = self else { return } let encoder = context.commandBuffer.makeBlitCommandEncoder() encoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture) encoder?.endEncoding() renderer.update(atTime: context.time) let descriptor = MTLRenderPassDescriptor() descriptor.colorAttachments[0].loadAction = .load descriptor.colorAttachments[0].storeAction = .store descriptor.colorAttachments[0].texture = context.targetColorTexture renderer.render(withViewport: CGRect(x: 0, y: 0, width: context.targetColorTexture.width, height: context.targetColorTexture.height), commandBuffer: context.commandBuffer, renderPassDescriptor: descriptor) } } }
Replies
9
Boosts
0
Views
2.9k
Activity
Apr ’26
ParticleEmitterComponent Position Offset Issue After iOS 26.1 Update – Seeking Solutions & Workarounds
Problem Summary After upgrading to iOS 26.1 and 26.2, I'm experiencing a particle positioning bug in RealityKit where ParticleEmitterComponent particles render at an incorrect offset relative to their parent entity. This behavior does not occur on iOS 18.6.2 or earlier versions, suggesting a regression introduced in the newer OS builds. Environment Details Operating System: iOS 26.1 & iOS 26.2 Framework: RealityKit Xcode Version: 16.2 (16C5032a) Expected vs. Actual Behavior Expected: Particles should render at the position of the entity to which the ParticleEmitterComponent is attached, matching the behavior on iOS 18.6.2 and earlier. Actual: Particles appear away from their parent entity, creating a visual misalignment that breaks the intended AR experience. Steps to Reproduce Create or open an AR application with RealityKit that uses particle components Attach a ParticleEmitterComponent to an entity via a custom system Run the application on iOS 26.1 or iOS 26.2 Observe that particles render at an offset position away from the entity Minimal Code Example Here's the setup from my test case: Custom Component & System: struct SparkleComponent4: Component {} class SparkleSystem4: System { static let query = EntityQuery(where: .has(SparkleComponent4.self)) required init(scene: Scene) {} func update(context: SceneUpdateContext) { for entity in context.scene.performQuery(Self.query) { // Only add once if entity.components.has(ParticleEmitterComponent.self) { continue } var newEmitter = ParticleEmitterComponent() newEmitter.mainEmitter.color = .constant(.single(.red)) entity.components.set(newEmitter) } } } AR Setup: let material = SimpleMaterial(color: .gray, roughness: 0.15, isMetallic: true) let model = Entity() model.components.set(ModelComponent(mesh: boxMesh, materials: [material])) model.components.set(SparkleComponent4()) model.position = [0, 0.05, 0] model.name = "MyCube" let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: [0.2, 0.2])) anchor.addChild(model) arView.scene.addAnchor(anchor) Questions for the Community Has anyone else encountered this particle positioning issue after updating to iOS 26.1/26.2? Are there known workarounds or configuration changes to ParticleEmitterComponent that restore correct positioning? Is this a confirmed bug, or could there be a change in coordinate system handling or transform inheritance that I'm missing? Additional Information I've already submitted this issue via Feedback Assistant(FB21346746)
Replies
3
Boosts
1
Views
1.6k
Activity
Apr ’26
RealityKit fill the background environment
I am new to RealityKit and Metal and I am building a RealityKit app that renders a procedural LowLevelMesh road. But the left and right side of the road is a complete green terrain mesh object and it doesn't look great. What I want is to add some rocks, tall trees and dence bushes (or weed) to make it look like the player is in the woods. But when I add many of those objects then the performance drains. What is the best approach to fill background empty spaces in the scene?
Replies
3
Boosts
0
Views
874
Activity
Apr ’26
Blending walk and run animations in RealityKit
Hi everybody, I have 2 separate animations run.usdz and walk.usdz animation files which are loaded perfectly in Reality Composer Pro and in the RealityKit application. I want to gradually increase the speed of my player by switching blending weight values from 0.0 (walking) to 1.0 (full speed running). let rabbit = await RabbitBuilder.loadWalkingRabbit() let runningRabbit = await RabbitBuilder.loadRunningRabbit() rabbit.scale = SIMD3(0.05, 0.05, 0.05) runningRabbit.scale = SIMD3(0.05, 0.05, 0.05) let walkAnimation = rabbit.availableAnimations let runAnimation = runningRabbit.availableAnimations RabbitWalker.walkAnim = walkAnimation.first! RabbitWalker.runAnim = runAnimation.first! guard let walk = RabbitWalker.walkAnim, let run = RabbitWalker.runAnim else { return } let blendTree = BlendTreeAnimation<JointTransforms>( BlendTreeBlendNode(sources: [ BlendTreeSourceNode(source: walk.definition, name: "walk", weight: .value(1 - weight)), BlendTreeSourceNode(source: run.definition, name: "run", weight: .value(weight)) ]), name: "rabbitLocomotion", repeatMode: .repeat, offset: TimeInterval(elapsed) ) // I have runtime error after executing this line: "Cannot add incompatible timeline type to blend tree." guard let resource = try? AnimationResource.generate(with: blendTree) else { return } entity.playAnimation(resource) static func loadWalkingRabbit() async -> Entity? { do { let scene = try await Entity(named: "Scene", in: realityKitEnvironmentBundle) guard let rabbit = await scene.findEntity(named: "RabbitWalk") else { return nil } await rabbit.removeFromParent() return rabbit } catch { return nil } } static func loadRunningRabbit() async -> Entity? { do { let scene = try await Entity(named: "Scene", in: realityKitEnvironmentBundle) guard let rabbit = await scene.findEntity(named: "RabbitRun") else { return nil } await rabbit.removeFromParent() return rabbit } catch { return nil } } But when I run this code I have this error; Cannot add incompatible timeline type to blend tree. By the way I have looked to developer's sample codes from here but I couldn't find any relevant BlendTreeAnimation sample which blends 2 animations. I would very happy if someone could direct me to a solution. Regards.
Replies
4
Boosts
0
Views
791
Activity
Apr ’26
Gestures not working correctly when setting the fov orientation to .horizontal
Hi there, I've discovered an issue with gesture handling in RealityKit when setting the camera’s fieldOfViewOrientation to horizontal. For instance, if I render a simple box at the center of the view with a collision shape that exactly matches its dimensions, the actual hit area behaves as if it's smaller than the box. Additionally, when attempting to drag the box away from the center, the hit area appears misaligned—offset slightly towards the center. Since the default fieldOfViewOrientation is vertical and everything works as expected in that mode, it seems that the gesture system might be assuming a vertical FOV. Given that the API allows setting it to horizontal, perhaps gestures should function correctly regardless of the orientation? Thank you!
Replies
2
Boosts
0
Views
1.2k
Activity
Apr ’26
Does VisionOS have the equivalent of ARView.physicsOrigin?
I'm trying to scale the physics of a scene without changing the apparent size to avoid the low-speed zeroing-out of motion that the physics simulation does. I found a technique for using separate simulation and physics roots in the docs, but it relies on ARView, which VisionOS doesn't have. This seems more elegant than scaling absolutely everything with shared root -- any chance I'm just failing in my searches to find the equivalent functionality?
Replies
3
Boosts
1
Views
1.1k
Activity
Mar ’26
RealityKit .Kinematic + collisions (visionOs)
Hi everyone, I'm new to visionOS development. I'm trying to create a physics-based scene (with gravity) where users can pick up and move objects on a workbench. I am struggling with physics interactions during the drag gesture: Kinematic Mode: If I switch to .kinematic during the drag, the object moves smoothly but clips through other objects (no collisions). Dynamic Mode: I tried keeping it .dynamic and applying linear velocity toward the hand position, but the movement feels laggy and unresponsive. Hybrid Approach: I tried switching to .kinematic during DragGesture.onChange and back to .dynamic on collision, but this causes the entity to jitter/shake violently when touching other objects. Has anyone found a clean way to drag objects while maintaining solid collisions. Thanks for your help!
Replies
3
Boosts
0
Views
1.2k
Activity
Mar ’26
RealityKit equivalent of ARGeoAnchor?
In ARKit there is ARGeoAnchor, which lets you anchor content using latitude and longitude so objects stay fixed to a real-world location. Is there an equivalent feature in RealityKit? I want to place points in the world and make sure they don't move or drift after placement. If RealityKit doesn't support this directly, what is the recommended approach?
Replies
0
Boosts
1
Views
864
Activity
Mar ’26