ARKit hand tracking

Hands on visionOS are represented as a series of anchors, and there are a lot of them! Each anchor has a transform. These anchors are located around the hand at positions like palm, wrist, index finger tip, etc.

There are two main ways we can work with these anchors.

Option 1: Anchoring Component (or AnchorEntity) + Spatial Tracking Session

This is a simple way to start working with the transforms for one or more AnchorEntities.

Start a Spatial Tracking Session. This will enable access to the transform of our anchors.

let configuration = SpatialTrackingSession.Configuration(
                    tracking: [.hand])
let session = SpatialTrackingSession()
await session.run(configuration)

Add an anchor. This example adds an anchor to the left hand index finger.

if let leftHandSphere = scene.findEntity(named: "LeftHand") {
  let leftHand = AnchorEntity(.hand(.left, location: .indexFingerTip))
  leftHand.addChild(leftHandSphere)
  content.add(leftHand)
}

Access the transform

let leftIndexTransform = Transform(matrix: anchor.transformMatrix(relativeTo: nil))

Important: if you are using collisions or physics, then you will also want to disable the default physics simulation for the anchor. Without this, the hand anchor won't be able to collide with other entities in the scene.

leftIndexAnchor.anchoring.physicsSimulation = .none

Option 2: Use ARKit directly

This is a bit more involved, but gives is more control. This example uses an ARKitSession and adds a sphere to each finger tip.

struct Example017: View {
    let arSession = ARKitSession()
    let handTrackingProvider = HandTrackingProvider()
    let leftCollection = Entity()
    let rightCollection = Entity()
    
    let tipJoints: [HandSkeleton.JointName] = [
        .thumbTip, .indexFingerTip, .middleFingerTip, .ringFingerTip, .littleFingerTip
    ]
    
    var body: some View {
        RealityView { content in
            content.add(leftCollection)
            content.add(rightCollection)
            
            if let scene = try? await Entity(named: "HandTrackingLabs", in: realityKitContentBundle) {
                content.add(scene)
                
                if let leftHandSphere = scene.findEntity(named: "StepSphereBlue") {
                    // Create clones of the left hand sphere for each joint
                    for jointName in tipJoints {
                        let sphere = leftHandSphere.clone(recursive: true)
                        sphere.name = jointName.description
                        leftCollection.addChild(sphere)
                    }
                    leftHandSphere.isEnabled = false
                }
                
                if let rightHandSphere = scene.findEntity(named: "StepSphereGreen") {
                    // Create clones of the right hand sphere for each joint
                    for jointName in tipJoints {
                        let sphere = rightHandSphere.clone(recursive: true)
                        sphere.name = jointName.description
                        rightCollection.addChild(sphere)
                    }
                    rightHandSphere.isEnabled = false
                }
            }
        }
        .persistentSystemOverlays(.hidden)
 
        .task { try! await arSession.run([handTrackingProvider]) }
 
        // Left Hand: Receive updates from the provider and process them over time
        .task {
            for await update in handTrackingProvider.anchorUpdates where update.anchor.chirality == .left {
                let handAnchor = update.anchor
                for jointName in tipJoints {
                    if let joint = handAnchor.handSkeleton?.joint(jointName),
                       let sphere = leftCollection.findEntity(named: jointName.description) {
                        let transform = handAnchor.originFromAnchorTransform 
                        let jointTransform = joint.anchorFromJointTransform
                        sphere.setTransformMatrix(transform * jointTransform, relativeTo: nil)
                    }
                }
            }
        }
        // Right Hand: Receive updates from the provider and process them over time
        .task {
            for await update in handTrackingProvider.anchorUpdates where update.anchor.chirality == .right {
                let handAnchor = update.anchor
                for jointName in tipJoints {
                    if let joint = handAnchor.handSkeleton?.joint(jointName),
                       let sphere = rightCollection.findEntity(named: jointName.description) {
                        let transform = handAnchor.originFromAnchorTransform 
                        let jointTransform = joint.anchorFromJointTransform
                        sphere.setTransformMatrix(transform * jointTransform, relativeTo: nil)
                    }
                }
            }
        }
    }
}

Resources

AnchorEntity https://developer.apple.com/documentation/realitykit/anchorentity

SpatialTrackingSession https://developer.apple.com/documentation/RealityKit/SpatialTrackingSession

ARKit Hand Tracking from Apple https://developer.apple.com/documentation/visionos/tracking-and-visualizing-hand-movement

I have several examples using AnchorEntity and Spatial Tracking Session on my site. https://stepinto.vision/learn-visionos/#hands