How do I freeze an AR scene in ARKit and then restore the scene elsewhere

Question

I have an AR scene (built as a node with children nodes) that I want the user to walk inside it. Because it can be large, I added a pause button in the corner so that if the user hits an obstacle in the real world, he'll just need to press-and-hold that button to freeze the scene, move elsewhere in the real world, and when the button will be released the scene would have moved as-is to that new location, retaining the same relative position and rotation inside that scene.

What I did is very simple. When button is touched down I just did: sceneView.session.pause()

And when the button is Touch Up Inside I ran: sceneView.session.run(configuration)

This seemed to do the trick.

However, after I clicked and moved once or twice more and moved back to where I was before, my whole AR scene suddenly jumped back to its previous location in the real world. So it did freeze and showed up in the new location for a second or two, but then it jumped back.

So I tried to resume the session with several of the available options. If I ran: sceneView.session.run(configuration, options: [.resetTracking])

the whole scene would move to the new location correctly as I wanted, but it would be set to its initial position in front of the camera (as opposed to being already "inside" the node) and rotation as if I was standing at (0,0,0) when I started the app, but now moved to the new location. In other words, it lost the current location and rotation that I was at when I pressed the pause button and just reset the scene.

All the other options that I tried had the same effect as if I didn't add any options.

I should add that I'm not using any anchors (so far) nor any ray casting. I didn't try different combinations of those options (except removeExistingAnchors and resetTracking together), but I doubt that that would help. Am I wrong?

What am I doing wrong? Anyone have a solution for my problem?

Or will I need to manually record the current position and rotation when pressing the pause button and then restore those with some position change functions of the node? I was hoping I could skip having to do it this way. And if I do need to do it manually, any tips on how to achieve this?

Thanks for your help!

484

Posted by

rafisaar

Reply

Add a Comment

Answer 1

You are seeing the effect of ARKit attempting to relocalize when the session is resumed. For your use case, you should not pause the session at all. What you essentially want to do is move AR content from one location to another (even if it is an entire hierarchy of nodes you are moving). The key to achieve that is raycasting and anchors. Basically you want to do the following:

Create an ARAnchor at the location where you initially position your scene's root node. As the user should be able to walk through the scene, it probably makes sense that this anchor is located on the floor. To find a suitable location for placing your content, you could turn on plane detection and then perform a raycast with an .existingPlane or .estimatedPlane target, and .horizontal alignment.
For "freezing" the scene, you could transform the anchor's position (in world coordinates) to camera coordinates, and then anchor your content to the camera. This will give you the effect that the scene is "frozen", i.e., does not move relative to the camera. Alternatively, you could continuously perform raycasts while the user keeps the button pressed, and use the raycast result for updating the position. This will still keep the content in front your camera, however with a "valid" placement on the floor.
For releasing the scene at the new location, you can do the same as in Step 1.

Posted by

Frameworks Engineer

Please see my further reply below. Thanks.

—
rafisaar

Add a Comment

Answer 2

You are seeing the effect of ARKit attempting to relocalize when the session is resumed. For your use case, you should not pause the session at all. What you essentially want to do is move AR content from one location to another (even if it is an entire hierarchy of nodes you are moving). The key to achieve that is raycasting and anchors. Basically you want to do the following:

Create an ARAnchor at the location where you initially position your scene's root node. As the user should be able to walk through the scene, it probably makes sense that this anchor is located on the floor. To find a suitable location for placing your content, you could turn on plane detection and then perform a raycast with an .existingPlane or .estimatedPlane target, and .horizontal alignment.
For "freezing" the scene, you could transform the anchor's position (in world coordinates) to camera coordinates, and then anchor your content to the camera. This will give you the effect that the scene is "frozen", i.e., does not move relative to the camera. Alternatively, you could continuously perform raycasts while the user keeps the button pressed, and use the raycast result for updating the position. This will still keep the content in front your camera, however with a "valid" placement on the floor.
For releasing the scene at the new location, you can do the same as in Step 1.

Posted by

Frameworks Engineer

Please see my further reply below. Thanks.

—
rafisaar

Add a Comment

Answer 3

Thanks a lot for this answer! I will need to learn this more in depth to understand how to transform these coordinates and how to anchor to the camera. What about rotation? Will I also need to retain the rotation so that the user will continue to point in the same direction relative to the scene? And for step 3, does that mean that I simply move the anchor again to the starting position relative to the current real position?

Thanks! That's very helpful.

Posted by

rafisaar

Add a Comment

Answer 4

Hi, I'm still struggling with this issue.

Following what was replied above in #2, I'm currently not using an anchor because I don't necessarily need to find a flat surface. Rather, my node is placed at a certain position relative to where we start at (0,0,0).

I have the following code which I'm still stuck with. When I add the node to the camera (pointOfView, last line of the code below), it does freeze in place, but I can't get it to freeze in the same position and orientation as it was before it was frozen.

@IBAction func pauseButtonClicked(_ sender: UIButton) {

    let currentPosition = sceneView.pointOfView?.position
    let currentEulerAngles = sceneView.pointOfView?.eulerAngles
    
    var internalNodeTraversal = lastNodeRootPosition - currentPosition!     // for now, lastNodeRootPosition is (0,0,0)         
    internalNodeTraversal.y = lastNodeRootPosition.y + 20       // just so it’s positioned a little higher in front of the camera

    myNode?.removeFromParentNode()      // remove the node from the Real World view.  Looks like this line has no effect and just adding the node as a child to the camera (pointOfView) is enough, but it feels more right to do this anyway.
    
    myNode?.position = internalNodeTraversal        // the whole node is moved respectively in the opposite direction from the root to where I’m standing to reposition the camera in my current position inside the node
    
 //       myNode?.eulerAngles = (currentEulerAngles! * -1)      — this code put the whole node in weird positions so I removed it
        myNode?.eulerAngles.y = currentEulerAngles!.y * -1  // opposite orientation of the node so the camera will be oriented in the same direction
        myNode?.eulerAngles.x = 0.3     // just tilting it up a little bit to have a better view, more similar to the view as before it was locked to the camera
// I don’t think I need to change the eulerAngles.z
        

    myNode!.convertPosition(internalNodeTraversal, to: sceneView.pointOfView)       // I’m not sure I wrote this correctly.  Also, this line doesn’t seem tp change anything
    
    sceneView.pointOfView?.addChildNode(myNode!)        // attaching the node to the camera so it will remain stuck while the user moves around until the button is released
}

So I first calculate where in the node I'm currently standing and then I change the position of the node in the opposite direction so that the camera will now be in that position. That seems to be correct. Now I need to change the orientation of the node so that it will point in the right direction and here things get funky. I've been trying so many things for days now. I use the eulerAngles for the orientation. If I set the whole vector multiplied by -1, it would show weird orientations. I ended up only using the eulerAngles.y which is the left/right orientation and I hardcoded the x orientation (up/down).

Ultimately what I have in the code above is the closest that I was able to get. If I'm pointing straight, the freeze will be correct. If I turn just a little bit, the freeze will be pretty close as well. Almost the same as what the user saw before the freeze. But the more I turn, the more the frozen image is off and more slanted. At some point (say I turn 50 or 60 degrees to the side) the whole node is off the camera and cannot be seen.

Somehow I have a feeling that there must be an easier and more correct way to achieve the above. In the reply above you wrote to "transform the anchor's position (in world coordinates) to camera coordinates". For that reason I added the "convertPosition" function in my code, but a) I'm not sure I used it correctly and b) it doesn't seem to change anything in my code if I have that line or not. Is that what you meant I should be doing at all?

What am I doing wrong? Any help would be very much appreciated. Thanks!

Posted by

rafisaar

Add a Comment

Answer 5

I found the solution. Actually, the problem I had was not even described as I didn't think it was relevant. I built the node 2 meters in front of the origin while the center of my node was still at the origin. So when I changed the rotation or eulerAngles, it rotated around the origin so my node moved in a large curve and in fact also changed its position as a result.

The solution was to use a pivot. Instead of changing the position and rotation of the node itself, I created a translation matrix and a rotation matrix which was at the point of the camera (where the user is standing) and I then multiplied both matrices. This would in effect freeze the node showing exactly what the user was seeing before it was frozen as the position is the same and the rotation is exactly around the user's standing position.

Posted by

rafisaar

Add a Comment

How do I freeze an AR scene in ARKit and then restore the scene elsewhere

Accepted Reply

Replies