Still don't understand why no one is clarifying about this Apple Video https://developer.apple.com/videos/play/wwdc2023/10111
At the end of this video, there's an incomplete tutorial about connecting a USDZ with mesh and Skeleton structure to the hand tracking system. No example project is linked, and no one is giving the community any clarification. Please can you help us to understand how to proceed?
We can create features like this using hand anchors. I can show you a few details on the RealityKit side, but creating the 3D hand assets would need to be done in Blender or some other 3D modeling app.
There are two ways to access these anchors.
-
Using ARKit: this is a good place to start
-
Using AnchorEntity or Anchoring Component to access hands. If all you need is attach visual items to the hands, this option is great. You don't need to request permission to use these anchors unless you want additional tracking data like transforms, physics, collisions.
Example: Create and customize entities for each primary location on the left hand
if let leftHandSphere = scene.findEntity(named: "StepSphereBlue") {
let indexTipAnchor = AnchorEntity(.hand(.left, location: .indexFingerTip), trackingMode: .continuous)
indexTipAnchor.addChild(leftHandSphere)
content.add(indexTipAnchor)
let palmAnchor = AnchorEntity(.hand(.left, location: .palm), trackingMode: .continuous)
palmAnchor.addChild(leftHandSphere.clone(recursive: true))
palmAnchor.position = [0, 0.05, 0]
palmAnchor.scale = [3, 3, 3]
content.add(palmAnchor)
let thumbTipAnchor = AnchorEntity(.hand(.left, location: .thumbTip), trackingMode: .continuous)
thumbTipAnchor.addChild(leftHandSphere.clone(recursive: true))
content.add(thumbTipAnchor)
let wristAnchor = AnchorEntity(.hand(.left, location: .wrist), trackingMode: .continuous)
wristAnchor.addChild(leftHandSphere.clone(recursive: true))
wristAnchor.scale = [3, 3, 3]
content.add(wristAnchor)
let aboveHandAnchor = AnchorEntity(.hand(.left, location: .aboveHand), trackingMode: .continuous)
aboveHandAnchor.addChild(leftHandSphere.clone(recursive: true))
aboveHandAnchor.scale = [2, 2, 2]
content.add(aboveHandAnchor)
}
Example: Create an entity for each joint anchor on the right hand
if let rightHandSphere = scene.findEntity(named: "StepSphereGreen") {
// In ARKit, joints are availble as a ENUM HandSkeleton.JointName.allCases
// But in RealityKit we are not so lucky. Create an array of all joints to iterate over.
let joints: [AnchoringComponent.Target.HandLocation.HandJoint] = [
.forearmArm,
.forearmWrist,
.indexFingerIntermediateBase,
.indexFingerIntermediateTip,
.indexFingerKnuckle,
.indexFingerMetacarpal,
.indexFingerTip,
.littleFingerIntermediateBase,
.littleFingerIntermediateTip,
.littleFingerKnuckle,
.littleFingerMetacarpal,
.littleFingerTip,
.middleFingerIntermediateBase,
.middleFingerIntermediateTip,
.middleFingerKnuckle,
.middleFingerMetacarpal,
.middleFingerTip,
.ringFingerIntermediateBase,
.ringFingerIntermediateTip,
.ringFingerKnuckle,
.ringFingerMetacarpal,
.ringFingerTip,
.thumbIntermediateBase,
.thumbIntermediateTip,
.thumbKnuckle,
.thumbTip,
.wrist
]
for joint in joints {
let anchor = AnchorEntity(
.hand(.right, location: .joint(for: joint)),
trackingMode: .continuous
)
anchor.addChild(rightHandSphere.clone(recursive: true))
anchor.position = rightHandSphere.position
content.add(anchor)
}
}