Hand Tracking Latency When UITextView Becomes Active in Vision Pro Immersive Space

Question

Created 3d

Replies 0

Boosts 0

Participants 1

I'm placing sphere at finger tip and updating its position as hand move. Finger joint tracking functions correctly, but I’ve observed noticeable latency in hand tracking updates whenever a UITextView becomes active. This lag happens intermittently during app usage, lasting about 5–10 seconds, after which the latency disappears and the sphere starts following the finger joints immediately.

When I open the immersive space for the first time, the profiler shows a large performance spike upto 328%. After that, it stabilizes and runs smoothly. Note: I don’t observe any lag when CPU usage spikes to 300% (upon immersive view load) yet the lag still occurs even when CPU usage remains below 100%.

I’m using the following code for hand tracking:


private func processHandTrackingUpdates() async {
    for await update in handTracking.anchorUpdates {
        let handAnchor = update.anchor
        if handAnchor.isTracked {
            switch handAnchor.chirality {
            case .left:
                leftHandAnchor = handAnchor
                updateHandJoints(for: handAnchor, with: leftHandJointEntities)
            case .right:
                rightHandAnchor = handAnchor
                updateHandJoints(for: handAnchor, with: rightHandJointEntities)
            }
        } else {
            switch handAnchor.chirality {
            case .left:
                leftHandAnchor = nil
                hideAllJoints(in: leftHandJointEntities)
            case .right:
                rightHandAnchor = nil
                hideAllJoints(in: rightHandJointEntities)
            }
        }
        
        await MainActor.run {
            handTrackingData.processNewHandAnchors(
                leftHand: self.leftHandAnchor,
                rightHand: self.rightHandAnchor
            )
        }
    }
}

And here’s the function I’m using to update the joint positions:

private func updateHandJoints(
    for handAnchor: HandAnchor,
    with jointEntities: [HandSkeleton.JointName: Entity]
) {
    guard handAnchor.isTracked else {
        hideAllJoints(in: jointEntities)
        return
    }
    
    // Check if the little finger tip and intermediate base are both tracked.
    if let tipJoint = handAnchor.handSkeleton?.joint(.littleFingerTip),
       let intermediateBaseJoint = handAnchor.handSkeleton?.joint(.littleFingerIntermediateTip),
       tipJoint.isTracked,
       intermediateBaseJoint.isTracked,
       let pinkySphere = jointEntities[.littleFingerTip] {
        
        // Convert joint transforms to world space.
        let tipTransform = handAnchor.originFromAnchorTransform * tipJoint.anchorFromJointTransform
        let intermediateBaseTransform = handAnchor.originFromAnchorTransform * intermediateBaseJoint.anchorFromJointTransform
        
        // Extract positions from the transforms.
        let tipPosition = SIMD3<Float>(tipTransform.columns.3.x,
                                       tipTransform.columns.3.y,
                                       tipTransform.columns.3.z)
        let intermediateBasePosition = SIMD3<Float>(intermediateBaseTransform.columns.3.x,
                                                    intermediateBaseTransform.columns.3.y,
                                                    intermediateBaseTransform.columns.3.z)
        
        // Calculate the midpoint.
        let midpointPosition = (tipPosition + intermediateBasePosition) / 2.0
        
        // Position the sphere at the midpoint and make it visible.
        pinkySphere.isEnabled = true
        pinkySphere.transform.translation = midpointPosition
        
    } else {
        // If either joint is not tracked, hide the sphere.
        jointEntities[.littleFingerTip]?.isEnabled = false
    }
    
    // Update the positions of all other hand joint spheres.
    for (jointName, entity) in jointEntities {
        if jointName == .littleFingerTip {
            // Already handled the pinky above.
            continue
        }
        
        guard let joint = handAnchor.handSkeleton?.joint(jointName),
              joint.isTracked else {
            entity.isEnabled = false
            continue
        }
        
        entity.isEnabled = true
        let jointTransform = handAnchor.originFromAnchorTransform * joint.anchorFromJointTransform
        entity.transform.translation = SIMD3<Float>(jointTransform.columns.3.x,
                                                    jointTransform.columns.3.y,
                                                    jointTransform.columns.3.z)
    }
}

I’ve attached both a profiler trace and a video recording from Vision Pro that clearly demonstrate the issue.

Profiler: https://drive.google.com/file/d/1fDWyGj_fgxud2ngkGH_IVmuH_kO-z0XZ

Vision Pro Recordings: https://drive.google.com/file/d/17qo3U9ivwYBsbaSm26fjaOokkJApbkz-

https://drive.google.com/file/d/1LxTxgudMvWDhOqKVuhc3QaHfY_1x8iA0

Has anyone else experienced this behavior? My thought is that there might be some background calculations happening at the OS level causing this latency. Any guidance would be greatly appreciated.

Thanks!

Boost