ARWorld loading works differently on iOS 15

There is a difference in how ARWorldMap loading/relocalization works on iOS 14 and 15, which is not mentioned in iOS 15 release notes.

Whenever you start an AR session by providing an initialWorldMap and relocalization succeeds, on iOS 14 the AR world's origin is automatically updated to match the previously saved world's origin. However, on iOS 15 it's not.

In my case, I've used RealityKit for rendering and found out about it by creating an AnchorEntity() (default identity transform) and adding it to my ARView's scene after relocalization. On iOS 14, it is placed where the previously saved session started, as expected. While on iOS 15, it is placed exactly where the current session started.

It's not the case for the Apple's sample: Saving and Loading World Data, because the virtual object is restored using ARSessionDelegate's method where a previously saved ARAnchor is provided by the session. This is also the solution I use right now – to add an anchor before saving the world so you can use it later to e.g. update the world's origin using setWorldOrigin(relativeTransform:).

I'm sharing in case someone runs into this problem, and it'd be great if someone at Apple could confirm that's the desired behavior.

Up vote post of bprzemyslaw Down vote post of bprzemyslaw
2.4k views
Posted by
bprzemyslaw
Reply

  • Thank you so much for this. This change has really broken my app, and I was having no luck at all figuring out how to address it. I will give this a shot.

    —
    paulruvolo

  • If I'm not mistaken, the transform of this ARAnchor that was added just before saving will not be updated after a relocalization until the moment it is in the camera's field of view again. So the after relocalization, the user should be made to point the camera at this ARAnchor and then when the ARSCNViewDelegate func didUpdate is called, we can use setWorldOrigin(relativeTransform:)?

    —
    MarnixArnold

  • (or maybe better session(_ session: ARSession, didUpdate anchors: [ARAnchor]) of ARSessionObserver, to not be dependent on SceneKit.)

    —
    MarnixArnold

Accepted Reply

Hi, if there is a difference in how your app runs between iOS 14 and iOS 15 without you using new APIs, then please file this as a bug on Feedback Assistant. Post the feedback number here, and if this is indeed expected behavior, the feedback item will get sent back to you. Then you can update this post with that info. My guess is this is a bug, but file the bug so it can get routed to ARKit for confirmation.

Posted by
Graphics and Games Engineer
Up vote reply of Graphics and Games Engineer Down vote reply of Graphics and Games Engineer
Post marked as solved

  • Okay. I submitted this as a feedback item. The workaround mentioned by bprzemyslaw seems to work, although the behavior is inconsistent when used in conjunction with a paused ARSession (so far I can only get it to work reliably when I don't pause the ARSession). The other issue I see is that the workaround only will work with maps that were saved with this special origin anchor. If the map doesn't have this anchor, the workaround doesn't seem to be applicable (I tried a few strategies, but none have panned out).

    —
    paulruvolo

  • Hello - we are also experiencing the issue as described in the original post, and have been forced to turn off our app's relocalization features. As @paulruvulo mentions, the nice workaround provided by the original poster does not apply if attempting to relocalize to a worldMap that has been captured before this breaking change.

    We've also filed a bug through Feedback Assistant - the feedback number is FB9688565.

    —
    spelman7
Add a Comment

Replies

Meant to post my response as a comment rather than an answer. I'd delete this if I could figure out how to do it.

Posted by
paulruvolo
Up vote reply of paulruvolo Down vote reply of paulruvolo
Add a Comment

Hi, if there is a difference in how your app runs between iOS 14 and iOS 15 without you using new APIs, then please file this as a bug on Feedback Assistant. Post the feedback number here, and if this is indeed expected behavior, the feedback item will get sent back to you. Then you can update this post with that info. My guess is this is a bug, but file the bug so it can get routed to ARKit for confirmation.

Posted by
Graphics and Games Engineer
Up vote reply of Graphics and Games Engineer Down vote reply of Graphics and Games Engineer
Post marked as solved

  • Okay. I submitted this as a feedback item. The workaround mentioned by bprzemyslaw seems to work, although the behavior is inconsistent when used in conjunction with a paused ARSession (so far I can only get it to work reliably when I don't pause the ARSession). The other issue I see is that the workaround only will work with maps that were saved with this special origin anchor. If the map doesn't have this anchor, the workaround doesn't seem to be applicable (I tried a few strategies, but none have panned out).

    —
    paulruvolo

  • Hello - we are also experiencing the issue as described in the original post, and have been forced to turn off our app's relocalization features. As @paulruvulo mentions, the nice workaround provided by the original poster does not apply if attempting to relocalize to a worldMap that has been captured before this breaking change.

    We've also filed a bug through Feedback Assistant - the feedback number is FB9688565.

    —
    spelman7
Add a Comment

I’d like to take the opportunity to reemphasize the purpose of ARAnchors: By adding a user ARAnchor you tell ARKit that you’re interested in that location, which is important in case of pose graph updates as ARKit’s internal map is growing. In order words, this helps ARKit to optimize world-tracking accuracy in the area around the anchor, so that virtual objects appear to stay in place relative to the real world. Therefore, even if your application doesn't use the save/load map API, it’s generally recommended to always add virtual content using ARAnchors, rather than simply placing it in world coordinates (see also the ARAnchor documentation).

For best practices on restoring virtual content after relocalizing to a saved ARWorldMap, please refer to this developer sample.

Posted by
Frameworks Engineer
Post not yet marked as solved Up vote reply of Frameworks Engineer Down vote reply of Frameworks Engineer

  • Thanks for this reminder. Quick question on that, in the past I had tried adding ARAnchors to coincide with the user's position at various points in time as they moved around an environment. I found that the transforms of these anchors were never updated during the tracking session (or maybe moved by a minuscule amount). Is there a way to use ARAnchors that increases the likelihood they will be moved during pose graph updates? From what I know about graph SLAM, It seems at a minimum you want to be able to communicate to ARKit that a particular ARAnchor represents the pose of the user at some point in time. In this case, if the graph SLAM ever shifts this position during its optimization, the ARAnchor could move accordingly. The issue I see is that there doesn't seem to be anyway to communicate to ARKit that the ARAnchor should represent the pose of the phone at some point in time (rather than just some arbitrary content at a particular location in space).

    Is there a way to use ARAnchors in a way that allows one to access the results of graph SLAM so that one can recover the pose of the phone at points other than the current frame (i.e., from some time point in the past)? Similarly, if I have some content that I'd like to place on a planar surface, what's the best practice with that? Should I create my own ARPlaneAnchor? Should I just store the coordinates of my object in the ARPlaneAnchor's coordinate system?

    Thanks in advance for any information. I have always struggled to access the results of the ARKit SLAM algorithm at the level of granularity that I'd like.

    Also, I'm having trouble adding line breaks, so my apologies on that.

    —
    paulruvolo

  • Conceptually there is no difference between an anchor representing the user’s or any other object’s position. Nothing should prevent you from creating anchors for the phone’s position at a certain point in time if you want to keep track of those (you could use the frame’s timestamp as the anchor’s name to identify them).

    To your second question regarding placement on planes: the best way would be to cast a ray with the desired target (say, horizontal planes) and then create an ARAnchor with the raycast result transform. You could even use a tracked raycast to automatically have your object’s position refined as ARKit updates planes. A helpful developer sample for best practices w.r.t. object placement on planes is this one: https://developer.apple.com/documentation/arkit/environmental_analysis/placing_objects_and_handling_3d_interaction.

    —
    Frameworks Engineer

  • Okay, thanks for these answers. Is it fair to say that ARKit takes into account when the ARAnchor was initially added to the session and uses this timestamp to figure out how much to adjust the ARAnchor's position as the ARWorldMap is refined?

    I did some testing with ARAnchors the other day and found that in iOS 15 they do move an appreciable amount when you rescan the same area (as you suggested).

    —
    paulruvolo

The issue appears to have been fixed in iOS 15.2. Nothing about it in the release notes but it looks like relocalization works like it used to, again.

Posted by
MarnixArnold
Post not yet marked as solved Up vote reply of MarnixArnold Down vote reply of MarnixArnold
Add a Comment