Synchronizing Multi-User AR Experiences on Apple Vision Pro

Question

Revin93 OP

Created Oct ’24

Replies 5

Boosts 0

Participants 3

Hello Developers,

I am currently in the initial planning stages of my bachelor thesis in computer science, where I will be developing an application in collaboration with a manufacturer of large-scale machinery. One of the core features I aim to implement is the ability for multiple Apple Vision Pro users to view the same object in augmented reality simultaneously, each from their respective positions relative to the object.

I am still exploring how best to achieve this feature. My initial approach involves designating one device as the host of a "room" within the application, allowing other users to join. If I can accurately determine the relative positions of all users to the host device, it should be possible to display the AR content correctly in terms of angle, size, and location for each user.

Despite my research, I haven't found much information on similar projects, and I would appreciate any insights or suggestions. Specifically, I am curious about common approaches for synchronizing AR experiences across multiple devices. Given that the Apple Vision Pro does not have a GPS sensor, I am also looking for alternative methods to precisely determine the positions of multiple devices relative to each other.

Any advice or shared experiences would be greatly appreciated!

Best regards, Revin

Answered by Vision Pro Engineer in 810153022

Hi @Revin93

Assuming your experience is view only and there's no shared interaction, this can be done with an AnchorEntity. An AnchorEntity allows you to position an entity relative to real world content. For your use case consider the image or object targets. Here's an example that attaches someEntity to an image in the real world:

 guard let url = Bundle.main.url(forResource: "marker", withExtension: "png") else {return}
let anchorEntity = AnchorEntity(.referenceImage(from: .init(url, physicalSize: [0.2032, 0.2032])))
 
anchorEntity.addChild(someEntity)
content.add(anchorEntity)

Boost

Answer 1

Vision Pro Engineer OP

Apple

Oct ’24

Accepted Answer

Hi @Revin93

Assuming your experience is view only and there's no shared interaction, this can be done with an AnchorEntity. An AnchorEntity allows you to position an entity relative to real world content. For your use case consider the image or object targets. Here's an example that attaches someEntity to an image in the real world:

 guard let url = Bundle.main.url(forResource: "marker", withExtension: "png") else {return}
let anchorEntity = AnchorEntity(.referenceImage(from: .init(url, physicalSize: [0.2032, 0.2032])))
 
anchorEntity.addChild(someEntity)
content.add(anchorEntity)

1

Answer 2

Vision Pro Engineer OP

Apple

Oct ’24

Hi @Revin93

I forgot to mention, Explore object tracking for visionOS is relevant to your goal. You can use image tracking to achieve your goal, but if you want a markerless solution, object tracking is the way to go.

1

Answer 3

Revin93 OP

Oct ’24

Thank you for your helpful insights. I did some research as well and discovered that ARKit includes a built-in feature for creating multi-user AR experiences, as outlined here: https://developer.apple.com/documentation/arkit/arkit_in_ios/creating_a_multiuser_ar_experience. Although I am not yet certain if this will be fully compatible with the Apple Vision Pro, I assume it will be, as the device also relies on ARKit for development. I will have more clarity on this once the development process begins.

Best regards, Revin

0

Answer 4

antoniocatala OP

Feb ’25

Have you been able to verify whether it is fully compatible with Apple Vision Pro?

Best Regards, Antonio

0

Answer 5

Revin93 OP

Mar ’25

Hi @antoniocatala

After exploring various options, I discovered that shared AR experiences are currently limited primarily to Apple devices (iPhones/iPads). I implemented an alternative solution, as suggested by @Vision Pro Engineer using ImageTracking with a marker image to establish a shared reference point across all devices. By leveraging the MultipeerConnectivity Framework, I was able to successfully share location data relative to this reference point between devices. The implementation works quite effectively, with the only limitation being that each user must initially scan the marker image to calibrate their AR experience.

0

	guard let url = Bundle.main.url(forResource: "marker", withExtension: "png") else {return}
	let anchorEntity = AnchorEntity(.referenceImage(from: .init(url, physicalSize: [0.2032, 0.2032])))

	anchorEntity.addChild(someEntity)
	content.add(anchorEntity)