Human Body joint tracking in VisionOS

Question

Created 3d

Replies 3

Boosts 0

Participants 2

The goal is to achieve precise joint tracking for clinical assessment. The Doctor is wearing the AVP and observing the Patients movement.

Do you have any recommended best practices for integrating real-time joint tracking and displaying them on the patient within visionOS?

We attempted to use VNHumanBodyPose3DObservation, which theoretically should work, but we are unable to display the detected joints in an Immersive Space for real-time validation. This makes it difficult for the doctor to ensure accurate tracking and if possible a photo or video of the Range of Motion assessment would be needed for the patient record.

Are there alternative methods to achieve precise real-time joint tracking without requiring main camera access (com.apple.developer.arkit.main-camera-access.allow)?

Boost

Answer 1

Vision Pro Engineer OP

Apple

3d

Hi @PrivacyNerd

No, there are no alternative methods to achieve precise real-time joint tracking without requiring main camera access.

Using main camera access with computer vision (like VNHumanBodyPose3DObservation) is an ideal way to track joints in 2D space, but since there's no supported way for you to obtain depth for a main camera pixel (at a high frequency), you will run into trouble mapping the 2D point on the main camera image to a 3D location in world space.This post has related information. It mentions 2 alternatives, neither of which will help you achieve your exact goal, but I'm sharing it so you have all the context:

Raycast with scene reconstruction: this will not work well for items that move. The latency is too high.
Position an entity on the projection plane: this will let you position an entity to align with the x/y position (when viewed through the left eye) over the passthrough feed, but the entity's z axis will always match the projection plane. EG: This is not full 3D.

To sum it up, I'm not aware of a supported way for you to anchor content to another person's joints in visionOS with the APIs currently available. If you'd like us to consider adding the necessary functionality, please file an enhancement request using Feedback Assistant. Once you file the request, please post the FB number here.

If you're not familiar with how to file enhancement requests, take a look at Bug Reporting: How and Why?

1

Answer 2

Vision Pro Engineer OP

Apple

3d

@PrivacyNerd

Addendum: Have you considered, displaying the main camera feed in a 2D window and overlaying 2D indicators? You can record the contents of the 2D window using RPScreenRecorder. Accessing the main camera shows how to display the main camera feed in a 2D window. I'll keep an eye on this thread; let me know if you decide to take this approach and I'll share some performance tips for displaying the main camera feed in a 2D window.

0

Answer 3

PrivacyNerd OP

1d

Thanks for your answer, I will look into the things you mentioned, and come back with questions.

1