Handling user-initiated re-centering in group immersive space?

Question

Created Oct ’24

Replies 4

Boosts 0

Participants 2

Hi, currently tinkering with a little shareplay app for the Vision Pro that allows people to facetime and shareplay to play with random 3d models (as well as move them around, which should sync the model positions for everyone in relative space).

When the users start their facetime call, then open the immersive space to see the 3d models, the models load in properly in context of the group immersive space's coordinate system, and moving the models reflects the new positions real-time for each participant.

The main issue comes if/when users use the digital crown to re-center their view. It appears to re-center the model and view, which is expected. However, it also seems to re-position the model/root entity to match the user's origin. Not sure if this is intentional or not, but this essentially makes it so that it "de-syncs" the model (so me moving the model next to someone does not reflect it 1:1 - it still moves properly, but the new "initial" position after re-centering makes it offset).

Is there a potential solution or work-around for this such that re-centering the view doesn't de-sync the model/entity's position?

Rough code for my RealityView component is below:

RealityView { content, attachments in
            content.add(appModel.originEntity)
            appModel.originEntity.addChild(appModel.modelContainerEntity)
            appModel.setInitialModelPosition()
            configureGestures(forModel: appModel.modelContainerEntity)
            configureToolbarAttachment(content: content, attachments: attachments)
        } update: { content, _ in
             // I have modified the Apple provided gesture components to
              // send the app model the new positions/rotations
              // as well as broadcast the position/rotation to shareplay participants
             
              // When user re-centers view, it seems to also re-position the model
              // so that its origin is at the local user's origin, rather than
              // the original origin
             
              // Can we receive a notification that user has re-centered view?
              // Or some other work-around?
            
            appModel.modelContainerEntity.setPosition(appModel.modelState.position, relativeTo: nil)
   
appModel.modelContainerEntity.setOrientation(.init(appModel.modelState.rotation3d), relativeTo: nil)
        } attachments: {
            Attachment(id: "customViewAttachment") {
                CustomView()
            }
        }
        .installGestures()

Please let me know if anything wasn't clear or if more information is needed. Thanks!

Boost

Answer 1

Vision Pro Engineer OP

Apple

Oct ’24

Hi @kvdcm

Take a look at the immersiveSpaceDisplacement environment variable. I suspect it might be what you're looking for, but I'd have to recreate the issue to be certain. Please let me know how it turns out!

1

Answer 2

kvdcm OP

Oct ’24

@Vision Pro Engineer

Thanks for replying, I was taking a look at that earlier today and it looked promising, but as I thought about it I wasn't exactly sure how to apply it. I know immersiveSpaceDisplacement can be used to calculate the user's origin (as if it were a solo immersive space) but if I place an entity in a shared immersive space's origin, it should appear in the same relative spot for everyone anyway right?

Would it potentially be an issue with placing the entity while users are still entering the group/shared immersive space? i.e. 2 users load in and the model has been placed at the same time and then a 3rd user loads into the space, the model might get de-synced there?

In theory though, how should I best use immersiveSpaceDisplacement to place entities such that they appear properly in relation to everyone?

Additional context as well, before I was more familiar with RealityView and RealityKit, I used to reload the RealityView entirely when swapping models, but now I simply add and remove children from a container entity that I keep reference to in my app model. The old implementation with constant reality view reloading actually did keep the entities in sync, so maybe there's something weird with how I keep track of the container entity?

Thanks again for the help!

0

Answer 3

Vision Pro Engineer OP

Apple

Oct ’24

Accepted Answer

@kvdcm

Thanks for the context. I’m less convinced immersiveSpaceDisplacement will solve this issue. Pressing the digital crown should not change the shared origin for the group activity and it shouldn’t change an entity’s transform relative to that origin.

Here's my understanding of the issue. Please let me know if I'm holding it wrong.

Your app allows participants in a group activity to manipulate entities in an immersive space. An entity’s transform (position, rotation, scale) should be consistently posititioned relatvie to all the participants. For example, given 2 participants, A and B, and a toy car, if the toy car is in front of participant A, participant B should see it in font of participant A. Initally this works, but when participant B walks acroos the room and presses the digital crown, the toy car no longer appears in front of participant A (it appears offset). Subsiquent moves to the toy car are synced, but are offset since pressing the digital crown offset them.

Suggestions to try if you haven't already:

The shared root seems like a likely culprit since a disparity (across participants) in its transform would cause a uniform shift of all its children. Are you sure the root's transform is consistent for all participants?
Be sure the participant is spatial before syncing their position.
Try simplifying the experience then slowly layer back in the complexity. Reduce things to a single entity with no shared parent. Does pressing the digital crown still cause the shift? Next add, syncing to one entity (no shared parent) and make the same observation.
Revisit this session and this article. Maybe it'll spark something.

If none of that works, please file a bug via feedback assistant with a project that recreates the issue in a focused manor and reply with a link to it.

1

Answer 4

kvdcm OP

Oct ’24

@Vision Pro Engineer Appreciate the suggestions - it seems that it was a combination of items mentioned in your 1st and 2nd suggestion.

How I set up this demo app is that I have some core app model state struct that keeps track of various states for use in SharePlay. immersiveSpaceDisplacement did give me the right idea in which it mentions that for a single person in immersive space, the origin is at their feet and when another person joins via SharePlay, the group immersive space that gets created will offset that origin to some shared relative location. This new shared origin is where the model is initially located. When a 3rd person joins, the shared origin is again displaced to account for this new participant, but I was not updating the transform/location for the participants, so the new participant was seeing the 3d model offset from where it should be.

I'll need to do some more testing to see if the core app model state needs to update the positions for the original 2 participants or the new participant, but for now I simply have it so that new participants joining will reset the model to the shared origin each time (and thus is a requirement for my feature to have everyone join first before collaborating on the 3d model).

Thanks again for your help!

1