VisionOS hands replacement inside an Immersive Space

Still don't understand why no one is clarifying about this Apple Video https://developer.apple.com/videos/play/wwdc2023/10111

At the end of this video, there's an incomplete tutorial about connecting a USDZ with mesh and Skeleton structure to the hand tracking system. No example project is linked, and no one is giving the community any clarification. Please can you help us to understand how to proceed?

Answered by radicalappdev in 825213022

We can create features like this using hand anchors. I can show you a few details on the RealityKit side, but creating the 3D hand assets would need to be done in Blender or some other 3D modeling app.

There are two ways to access these anchors.

  1. Using ARKit: this is a good place to start

  2. Using AnchorEntity or Anchoring Component to access hands. If all you need is attach visual items to the hands, this option is great. You don't need to request permission to use these anchors unless you want additional tracking data like transforms, physics, collisions.

Example: Create and customize entities for each primary location on the left hand

if let leftHandSphere = scene.findEntity(named: "StepSphereBlue") {
    let indexTipAnchor = AnchorEntity(.hand(.left, location: .indexFingerTip), trackingMode: .continuous)
    indexTipAnchor.addChild(leftHandSphere)
    content.add(indexTipAnchor)

    let palmAnchor = AnchorEntity(.hand(.left, location: .palm), trackingMode: .continuous)
    palmAnchor.addChild(leftHandSphere.clone(recursive: true))
    palmAnchor.position =  [0, 0.05, 0]
    palmAnchor.scale = [3, 3, 3]
    content.add(palmAnchor)

    let thumbTipAnchor = AnchorEntity(.hand(.left, location: .thumbTip), trackingMode: .continuous)
    thumbTipAnchor.addChild(leftHandSphere.clone(recursive: true))
    content.add(thumbTipAnchor)

    let wristAnchor = AnchorEntity(.hand(.left, location: .wrist), trackingMode: .continuous)
    wristAnchor.addChild(leftHandSphere.clone(recursive: true))
    wristAnchor.scale = [3, 3, 3]
    content.add(wristAnchor)

    let aboveHandAnchor = AnchorEntity(.hand(.left, location: .aboveHand), trackingMode: .continuous)
    aboveHandAnchor.addChild(leftHandSphere.clone(recursive: true))
    aboveHandAnchor.scale = [2, 2, 2]
    content.add(aboveHandAnchor)
}

Example: Create an entity for each joint anchor on the right hand

if let rightHandSphere = scene.findEntity(named: "StepSphereGreen") {
    // In ARKit, joints are availble as a ENUM HandSkeleton.JointName.allCases
    // But in RealityKit we are not so lucky. Create an array of all joints to iterate over.
    let joints: [AnchoringComponent.Target.HandLocation.HandJoint] = [
        .forearmArm,
        .forearmWrist,
        .indexFingerIntermediateBase,
        .indexFingerIntermediateTip,
        .indexFingerKnuckle,
        .indexFingerMetacarpal,
        .indexFingerTip,
        .littleFingerIntermediateBase,
        .littleFingerIntermediateTip,
        .littleFingerKnuckle,
        .littleFingerMetacarpal,
        .littleFingerTip,
        .middleFingerIntermediateBase,
        .middleFingerIntermediateTip,
        .middleFingerKnuckle,
        .middleFingerMetacarpal,
        .middleFingerTip,
        .ringFingerIntermediateBase,
        .ringFingerIntermediateTip,
        .ringFingerKnuckle,
        .ringFingerMetacarpal,
        .ringFingerTip,
        .thumbIntermediateBase,
        .thumbIntermediateTip,
        .thumbKnuckle,
        .thumbTip,
        .wrist
    ]

    for joint in joints {
        let anchor = AnchorEntity(
            .hand(.right, location: .joint(for: joint)),
            trackingMode: .continuous
        )
        anchor.addChild(rightHandSphere.clone(recursive: true))
        anchor.position = rightHandSphere.position
        content.add(anchor)
    }
}

We can create features like this using hand anchors. I can show you a few details on the RealityKit side, but creating the 3D hand assets would need to be done in Blender or some other 3D modeling app.

There are two ways to access these anchors.

  1. Using ARKit: this is a good place to start

  2. Using AnchorEntity or Anchoring Component to access hands. If all you need is attach visual items to the hands, this option is great. You don't need to request permission to use these anchors unless you want additional tracking data like transforms, physics, collisions.

Example: Create and customize entities for each primary location on the left hand

if let leftHandSphere = scene.findEntity(named: "StepSphereBlue") {
    let indexTipAnchor = AnchorEntity(.hand(.left, location: .indexFingerTip), trackingMode: .continuous)
    indexTipAnchor.addChild(leftHandSphere)
    content.add(indexTipAnchor)

    let palmAnchor = AnchorEntity(.hand(.left, location: .palm), trackingMode: .continuous)
    palmAnchor.addChild(leftHandSphere.clone(recursive: true))
    palmAnchor.position =  [0, 0.05, 0]
    palmAnchor.scale = [3, 3, 3]
    content.add(palmAnchor)

    let thumbTipAnchor = AnchorEntity(.hand(.left, location: .thumbTip), trackingMode: .continuous)
    thumbTipAnchor.addChild(leftHandSphere.clone(recursive: true))
    content.add(thumbTipAnchor)

    let wristAnchor = AnchorEntity(.hand(.left, location: .wrist), trackingMode: .continuous)
    wristAnchor.addChild(leftHandSphere.clone(recursive: true))
    wristAnchor.scale = [3, 3, 3]
    content.add(wristAnchor)

    let aboveHandAnchor = AnchorEntity(.hand(.left, location: .aboveHand), trackingMode: .continuous)
    aboveHandAnchor.addChild(leftHandSphere.clone(recursive: true))
    aboveHandAnchor.scale = [2, 2, 2]
    content.add(aboveHandAnchor)
}

Example: Create an entity for each joint anchor on the right hand

if let rightHandSphere = scene.findEntity(named: "StepSphereGreen") {
    // In ARKit, joints are availble as a ENUM HandSkeleton.JointName.allCases
    // But in RealityKit we are not so lucky. Create an array of all joints to iterate over.
    let joints: [AnchoringComponent.Target.HandLocation.HandJoint] = [
        .forearmArm,
        .forearmWrist,
        .indexFingerIntermediateBase,
        .indexFingerIntermediateTip,
        .indexFingerKnuckle,
        .indexFingerMetacarpal,
        .indexFingerTip,
        .littleFingerIntermediateBase,
        .littleFingerIntermediateTip,
        .littleFingerKnuckle,
        .littleFingerMetacarpal,
        .littleFingerTip,
        .middleFingerIntermediateBase,
        .middleFingerIntermediateTip,
        .middleFingerKnuckle,
        .middleFingerMetacarpal,
        .middleFingerTip,
        .ringFingerIntermediateBase,
        .ringFingerIntermediateTip,
        .ringFingerKnuckle,
        .ringFingerMetacarpal,
        .ringFingerTip,
        .thumbIntermediateBase,
        .thumbIntermediateTip,
        .thumbKnuckle,
        .thumbTip,
        .wrist
    ]

    for joint in joints {
        let anchor = AnchorEntity(
            .hand(.right, location: .joint(for: joint)),
            trackingMode: .continuous
        )
        anchor.addChild(rightHandSphere.clone(recursive: true))
        anchor.position = rightHandSphere.position
        content.add(anchor)
    }
}

@radicalappdev thanks for the very helpful tip!

@SteveZ1982 I agree, a sample that shows how to replace a person's hands with virtual hands would be very useful. If you'd like us to consider creating one, please file an enhancement request using Feedback Assistant. Once you file the request, please post the FB number here.

If you're not familiar with how to file enhancement requests, take a look at Bug Reporting: How and Why?

VisionOS hands replacement inside an Immersive Space
 
 
Q