visionOS 3d interactions like the native keyboard when no longer observed in passthrough

Question

Created 2d

Replies 1

Boosts 0

Participants 2

While using apple's vision pro, we noticed that we can continue to use the visionOS keyboard when we no longer actually see it in passthrough.

In other words, when we focus on a field to type, visionOS displays the keyboard for us in such a way that we actually see it. Then, we noticed if we look away a little bit, either up, or down, or left, or right, in such a way that the keyboard is no longer visible by us in the passthrough, the keyboard still remains responsive to taps from our fingers at the location where it is. It seems the keyboard remains functional and responsive to taps even though we can no longer observe/see it.

We are trying to figure out how to implement similar functionality in our app whereby the user can continue to manipulate a 3d entity when the user can no longer actually observe it in passthrough (like the visionOS keyboard appears to allow).

I assume the visionOS keyboard has this functionality thanks to the downward facing sensors on the hardware that allow hand tracking even though the hands can no longer be observed by the user. That is likely how we can rest our hands on our lap is still be able to interact with visionOS.

How can we implement a similar functionality for 3D entities?
Is there a way to tap in, or to allow hand tracking, from those toward facing cameras?
Is it possible to manipulate a 3D entity when it is no longer observed by the user for example when they shift their attention somewhere else in the field of vision?
How does the visionOS keyboard achieve this?

Boost

Answer 1

Vision Pro Engineer OP

Apple

1d

Hi @VaiStardom

I can confirm SpatialTapGesture doesn't consistently register direct input (trigger callbacks) on entities outside a person's field of view. However, you can use hand tracking to create a custom gesture that works on all entities, regardless of their position, as long as visionOS can track the person's hands.

This approach only works within an ImmersiveSpace and requires adding an NSHandsTrackingUsageDescription entry to your app's Info.plist that explains why your app needs hand tracking access.

The code snippet below demonstrates this technique by playing a sound when a person taps a cube, even when it's outside their field of view.

import RealityKit
import SwiftUI
import AudioToolbox

struct ImmersiveView: View {
    @State var spatialTrackingSession = SpatialTrackingSession()
    @State var collisionEventSubscription: EventSubscription?
    
    var body: some View {
        RealityView { content in
            let cube = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .green, isMetallic: false)]
            )

            cube.generateCollisionShapes(recursive: false)
            cube.position = [0, 1.2, -0.5]
            content.add(cube)
            
            let indexFingerAnchor = AnchorEntity(
                .hand(.right, location: .indexFingerTip)
            )

            indexFingerAnchor.components.set(
                CollisionComponent(shapes: [.generateSphere(radius: 0.01)])
            )
            indexFingerAnchor.anchoring.physicsSimulation = .none
            content.add(indexFingerAnchor)

            collisionEventSubscription = content.subscribe(
                to: CollisionEvents.Ended.self,
                on: nil
            ) {
                event in
                if event.entityA == cube {
                    AudioServicesPlaySystemSound(1016)
                }
            }
        }
        .task {
            // Start a spatial tracking session so our hand anchor entities can receive collisions.

            let configuration = SpatialTrackingSession.Configuration(
                tracking: [.hand])
            await spatialTrackingSession.run(configuration)
        }
    }
}

Consider filing an enhancement request

I'd you'd like us to consider abstracting this behavior, please file a feedback request and explain how your use case would benefit from the abstraction. Once you file the request, please post the FB number here.

If you're not familiar with how to file enhancement requests, take a look at Bug Reporting: How and Why?

0