You can use gestures in your systems. Apple has a couple of example projects that show some methods for creating components and systems that use SwiftUI gestures Transforming RealityKit entities using gestures https://developer.apple.com/documentation/realitykit/transforming-realitykit-entities-with-gestures While not the focus of the example, this particle example also has an interesting system. Simulating particles in your visionOS app https://developer.apple.com/documentation/realitykit/simulating-particles-in-your-visionos-app Both of these have been helpful in learning how to use gestures from within a system.

We can use ARKit hand tracking or use AnchorEntity with SpatialTrackingSession. Here is an example with SpatialTrackingSession. This adds some anchors to the users hands, then enables those anchors to collide with other entities in the scene. Once you detect the collisions, you can execute some code to show your window or attachment. Spatial Tracking Session Anchor Entity Important: make sure to set this value to none of the anchor will not be able to interact with other entities. leftIndexAnchor.anchoring.physicsSimulation = .none This example uses trigger collisions instead of physics. The entities were created in Reality Composer Pro, then loaded in the RealityView. struct Example021: View { var body: some View { RealityView { content in if let scene = try? await Entity(named: "HandTrackingLabs", in: realityKitContentBundle) { content.add(scene) // 1. Set up a Spatial Tracking Session with hand tracking. // This will add ARKit features to our Anchor Entities, enabling collisions. let configuration = SpatialTrackingSession.Configuration( tracking: [.hand]) let session = SpatialTrackingSession() await session.run(configuration) if let subject = scene.findEntity(named: "StepSphereRed"), let stepSphereBlue = scene.findEntity(named: "StepSphereBlue"), let stepSphereGreen = scene.findEntity(named: "StepSphereGreen") { content.add(subject) // 2. Create an anchor for the left index finger let leftIndexAnchor = AnchorEntity(.hand(.left, location: .indexFingerTip), trackingMode: .continuous) // 3. Disable the default physics simulation on the anchor leftIndexAnchor.anchoring.physicsSimulation = .none // 4. Add the sphere to the anchor and add the anchor to the scene graph leftIndexAnchor.addChild(stepSphereBlue) content.add(leftIndexAnchor) // Repeat the same steps for the right index finger let rightIndexAnchor = AnchorEntity(.hand(.right, location: .indexFingerTip), trackingMode: .continuous) rightIndexAnchor.anchoring.physicsSimulation = .none // rightIndexAnchor.addChild(stepSphereGreen) content.add(rightIndexAnchor) // Example 1: Any entity can collide with any entity. Fire a particle burst // Allow collision between the hand anchors // Allow collision between a hand anchor and the subject _ = content.subscribe(to: CollisionEvents.Began.self) { collisionEvent in print("Collision unfiltered \(collisionEvent.entityA.name) and \(collisionEvent.entityB.name)") collisionEvent.entityA.components[ParticleEmitterComponent.self]?.burst() } // Example 2: Only track collisions on the subject. Swap the color of the material based on left or right hand. _ = content .subscribe(to: CollisionEvents.Began.self, on: subject) { collisionEvent in print("Collision Subject Color Change \(collisionEvent.entityA.name) and \(collisionEvent.entityB.name)") if(collisionEvent.entityB.name == "StepSphereBlue") { swapColorEntity(subject, color: .stepBlue) } else if (collisionEvent.entityB.name == "StepSphereGreen") { swapColorEntity(subject, color: .stepGreen) } } } } } } func swapColorEntity(_ entity: Entity, color: UIColor) { if var mat = entity.components[ModelComponent.self]?.materials.first as? PhysicallyBasedMaterial { mat.baseColor = .init(tint: color) entity.components[ModelComponent.self]?.materials[0] = mat } } }

I found an alternative method for this in the particle example project. Instead of using value.gestureValue.translation3D to move the entity, this version uses value.location3D and value.startLocation3D. It’s not quite as good as the gesture Apple uses on Windows and Volumes. However, it is far better than what I’ve been using until now. I'd love to hear any ideas for how to improve this struct Example046: View { var body: some View { RealityView { content in if let scene = try? await Entity(named: "GestureLabs", in: realityKitContentBundle) { content.add(scene) // Lower the entire scene to the bottom of the volume scene.position = [1, 1, -1.5] } } .modifier(DragGestureWithPivot046()) } } fileprivate struct DragGestureWithPivot046: ViewModifier { @State var isDragging: Bool = false @State var initialPosition: SIMD3<Float> = .zero func body(content: Content) -> some View { content .gesture( DragGesture() .targetedToAnyEntity() .onChanged { value in // We we start the gesture, cache the entity position if !isDragging { isDragging = true initialPosition = value.entity.position } guard let entityParent = value.entity.parent else { return } // The current location: where we are in the gesture let gesturePosition = value.convert(value.location3D, from: .global, to: entityParent) // Minus the start location of the gesture let deltaPosition = gesturePosition - value.convert(value.startLocation3D, from: .global, to: entityParent) // Plus the initial position of the entity let newPos = initialPosition + deltaPosition // Optional: using move(to:) to smooth out the movement let newTransform = Transform( scale: value.entity.scale, rotation: value.entity.orientation, translation: newPos ) value.entity.move(to: newTransform, relativeTo: entityParent, duration: 0.1) // Or set the position directly // value.entity.position = newPos } .onEnded { value in // Clean up when the gesture has ended isDragging = false initialPosition = .zero } ) } }

Are you using scene phase in the extra window too? You have to implemented separately in each window. The code above only showed it in the MyScene window. I like to set up a central bit of state to track my the open status of my scenes. I made an example of this a while back. Hope it helps! https://github.com/radicalappdev/Step-Into-Example-Projects/tree/main/Garden06

We can create features like this using hand anchors. I can show you a few details on the RealityKit side, but creating the 3D hand assets would need to be done in Blender or some other 3D modeling app. There are two ways to access these anchors. Using ARKit: this is a good place to start Using AnchorEntity or Anchoring Component to access hands. If all you need is attach visual items to the hands, this option is great. You don't need to request permission to use these anchors unless you want additional tracking data like transforms, physics, collisions. Example: Create and customize entities for each primary location on the left hand if let leftHandSphere = scene.findEntity(named: "StepSphereBlue") { let indexTipAnchor = AnchorEntity(.hand(.left, location: .indexFingerTip), trackingMode: .continuous) indexTipAnchor.addChild(leftHandSphere) content.add(indexTipAnchor) let palmAnchor = AnchorEntity(.hand(.left, location: .palm), trackingMode: .continuous) palmAnchor.addChild(leftHandSphere.clone(recursive: true)) palmAnchor.position = [0, 0.05, 0] palmAnchor.scale = [3, 3, 3] content.add(palmAnchor) let thumbTipAnchor = AnchorEntity(.hand(.left, location: .thumbTip), trackingMode: .continuous) thumbTipAnchor.addChild(leftHandSphere.clone(recursive: true)) content.add(thumbTipAnchor) let wristAnchor = AnchorEntity(.hand(.left, location: .wrist), trackingMode: .continuous) wristAnchor.addChild(leftHandSphere.clone(recursive: true)) wristAnchor.scale = [3, 3, 3] content.add(wristAnchor) let aboveHandAnchor = AnchorEntity(.hand(.left, location: .aboveHand), trackingMode: .continuous) aboveHandAnchor.addChild(leftHandSphere.clone(recursive: true)) aboveHandAnchor.scale = [2, 2, 2] content.add(aboveHandAnchor) } Example: Create an entity for each joint anchor on the right hand if let rightHandSphere = scene.findEntity(named: "StepSphereGreen") { // In ARKit, joints are availble as a ENUM HandSkeleton.JointName.allCases // But in RealityKit we are not so lucky. Create an array of all joints to iterate over. let joints: [AnchoringComponent.Target.HandLocation.HandJoint] = [ .forearmArm, .forearmWrist, .indexFingerIntermediateBase, .indexFingerIntermediateTip, .indexFingerKnuckle, .indexFingerMetacarpal, .indexFingerTip, .littleFingerIntermediateBase, .littleFingerIntermediateTip, .littleFingerKnuckle, .littleFingerMetacarpal, .littleFingerTip, .middleFingerIntermediateBase, .middleFingerIntermediateTip, .middleFingerKnuckle, .middleFingerMetacarpal, .middleFingerTip, .ringFingerIntermediateBase, .ringFingerIntermediateTip, .ringFingerKnuckle, .ringFingerMetacarpal, .ringFingerTip, .thumbIntermediateBase, .thumbIntermediateTip, .thumbKnuckle, .thumbTip, .wrist ] for joint in joints { let anchor = AnchorEntity( .hand(.right, location: .joint(for: joint)), trackingMode: .continuous ) anchor.addChild(rightHandSphere.clone(recursive: true)) anchor.position = rightHandSphere.position content.add(anchor) } }

My understanding is that those cameras are not relevant for visionOS development. Those are used on iOS and other platforms. When I tried to do this last winter it seemed that the only answer was to move the world around the user, instead of moving the user around the world. Here is an excerpt from from the code I came up with. This allows as user to tap on an entity and move to a new position. Sort of like "waypoint" teleportation that was common in VR games circa 2016-2017. This could be improved in lots of ways. For example, using SpatialEventGesture or SpatialTapGesture to get a more precise location. struct Lab5017: View { @State var selected: String = "Tap Something" @State var sceneContent: Entity? @State var sceneContentPosition: SIMD3<Float> = [0,0,0] var tap: some Gesture { SpatialTapGesture() .targetedToAnyEntity() .onEnded { value in selected = value.entity.name // Calculate the vector from the origin to the tapped position let vectorToTap = value.entity.position // Normalize the vector to get a direction from the origin to the tapped position let direction = normalize(vectorToTap) // Calculate the distance (or magnitude) between the origin and the tapped position let distance = length(vectorToTap) // Calculate the new position by inverting the direction multiplied by the distance let newPosition = -direction * distance // Update sceneOffset's X and Z components, leave Y as it is sceneContentPosition.x = newPosition.x sceneContentPosition.z = newPosition.z } } var body: some View { RealityView { content, attachments in if let model = try? await Entity(named: "5017Move", in: realityKitContentBundle) { content.add(model) // Get the scene content and stash it in state if let floorParent = model.findEntity(named: "SceneContent") { sceneContent = floorParent sceneContentPosition = floorParent.position } } //Position the attachment somewhere we can see it if let attachmentEntity = attachments.entity(for: "SelectedLabel") { attachmentEntity.position = [0.8, 1.5, -2] attachmentEntity.scale = [5,5,5] content.add(attachmentEntity) } } update: { content, attachments in // Update the position of scene content anytime we get a new position sceneContent?.position = sceneContentPosition } attachments: { Attachment(id: "SelectedLabel") { Text(selected) .font(.largeTitle) .padding(18) .background(.black) .cornerRadius(12) } } .gesture(tap) // The floor child entities can receive input, so this gesture will fire when we tap them } }

By default, these dynamic lights will not affect the passthrough environment. But there is an interesting workaround that we can use. We can use an entity either a shader graph material that uses the "ShadowReceivingOcclusionSurface" node. This small demo scene has a 1x0.1x1 cube that is using that material. Then I dropped in a spotlight and some other cubes to block the light. The spotlight can shine only the surface of the object using the occlusion material. This works with shadows cast by virtual objects too, and it doesn't require the grounding shadow component. The challenge will be determining what mesh/object to use for the occlusion material. Depending on your use case, simple shapes may work. Or you may want to use planes or the room mesh from ARKit.

@Vision Pro Engineer Hi, thanks for the response. I have a few questions and responses it's usually better to put the code that configures the SpatialTrackingSession into a class instead of an @State property on your view Sure, I would do that in most apps. This is just an example where I was trying to keep everything in one file. Do you have any details on the why it is better to place SpatialTrackingSession in an observable class instead of state on a view? Several of the WWDC sessions and examples store the session in the view and I was using them as a starting point. SpatialTapGesture Just to clarify, that was just an example from "Deep dive into volumes and immersive spaces" (WWDC 2024). They showed using the tap gesture on an anchor but didn't show how the collision was created. I wasn't using a gesture in my scene at all. I was just using this as an example of something that obviously needed a collision shape. But the session obscured the details. this CollisionComponent(shapes: .init()) is creating a collision component with an empty array Yes, I was trying to create a collision, the populate it later with a call to generateCollisionShapes event.anchor.generateCollisionShapes(recursive: true) If I understand correctly this doesn't work because the AnchorEntity is a point on a plane, not a the plane it self. Is that correct? Your hard coded ShapeResouce is interesting, but it doesn't help me create a collision shape that matches the physical floor in my office. This results in an arbitrary shape, positioned by a system I can't predict, to create a floor that may or may not cover the floor in the real room. Is it possible to use an AnchorEntity (with SpatialTrackingSession) to get the plane/bounds/rect of the floor that visionOS detected? So far my guess is no. It seems like AnchorEntity is actually an arbitrary point/transform on that detected plane.

Jessy on Bluesky answered this. Apparently components, can only use literals for their initial values https://bsky.app/profile/jessymeow.bsky.social/post/3lcteyvd6422o public var BoxDimensions: SIMD3<Float> = [2.0, 2.0, 2.0] public var PlaneDimensions: SIMD2<Float> = [2.0, 2.0]

I haven't found a way to gain access to the camera or player entity from the context of RealityView. In the meantime, I put together a quick demo that moves content in a USDA scene to the player. Move the world instead of the player. struct Lab5017: View { @State var selected: String = "Tap Something" @State var sceneContent: Entity? @State var sceneContentPosition: SIMD3<Float> = [0,0,0] var tap: some Gesture { SpatialTapGesture() .targetedToAnyEntity() .onEnded { value in selected = value.entity.name // Calculate the vector from the origin to the tapped position let vectorToTap = value.entity.position // Normalize the vector to get a direction from the origin to the tapped position let direction = normalize(vectorToTap) // Calculate the distance (or magnitude) between the origin and the tapped position let distance = length(vectorToTap) // Calculate the new position by inverting the direction multiplied by the distance let newPosition = -direction * distance // Update sceneOffset's X and Z components, leave Y as it is sceneContentPosition.x = newPosition.x sceneContentPosition.z = newPosition.z } } var body: some View { RealityView { content, attachments in if let model = try? await Entity(named: "5017Move", in: realityKitContentBundle) { content.add(model) // Get the scene content and stash it in state if let floorParent = model.findEntity(named: "SceneContent") { sceneContent = floorParent sceneContentPosition = floorParent.position } } //Position the attachment somewhere we can see it if let attachmentEntity = attachments.entity(for: "SelectedLabel") { attachmentEntity.position = [0.8, 1.5, -2] attachmentEntity.scale = [5,5,5] content.add(attachmentEntity) } } update: { content, attachments in // Update the position of scene content anytime we get a new position sceneContent?.position = sceneContentPosition } attachments: { Attachment(id: "SelectedLabel") { Text(selected) .font(.largeTitle) .padding(18) .background(.black) .cornerRadius(12) } } .gesture(tap) // The floor child entities can receive input, so this gesture will fire when we tap them } }