Sample Code

Placing Objects and Handling 3D Interaction

Place virtual content on real-world surfaces, and enable the user to interact with virtual content by using gestures.

Download

Overview

The key facet of an AR experience is the ability to intermix virtual and real-world objects. A flat surface is the optimum location for setting a virtual object. To assist ARKit with finding surfaces, you tell the user to move their device in ways that help ARKit initialize the session. ARKit provides a view that tailors its instructions to the user for you, guiding them to the surface that your app needs.

To enable the user to put a virtual item on the real-world surface when they tap the screen, ARKit incorporates ray casting, which provides a 3D location in physical space that corresponds to the screen’s touch location. If the user wants to rotate or otherwise move the virtual items they place, you respond to the respective touch gestures and correlate that input to the virtual content’s look in the physical environment.

Set a Goal to Coach the User’s Movement

To enable your app to detect real-world surfaces, you use a world tracking configuration. For ARKit to establish tracking, the user must physically move their device to allow ARKit to get a sense of perspective. To communicate this need to the user, you use a view provided by ARKit that presents the user with instructional diagrams and verbal guidance, called ARCoachingOverlayView. For example, when you start the app, the first thing the user sees is a message and animation from the coaching overlay telling them to move their device left and right, repeatedly, in order to get started.

To enable the user to place virtual content on a horizontal surface, you set the coaching overlay goal accordingly.

func setGoal() {
    coachingOverlay.goal = .horizontalPlane
}

The coaching overlay then tailors its instructions according to the goal you choose. After ARKit gets a sense of perspective, the coaching overlay instructs the user to find a surface.

Respond to Coaching Events

To make sure the coaching overlay provides guidance to the user whenever ARKit determines it’s necessary, you set activatesAutomatically to true.

func setActivatesAutomatically() {
    coachingOverlay.activatesAutomatically = true
}

The coaching overlay activates automatically when the app starts, or when tracking degrades past a certain threshold. In those situations, ARKit notifies your delegate by calling coachingOverlayViewWillActivate(_:). In response to this event, hide your app’s UI to enable the user to focus on the instructions that the coaching overlay provides.

func coachingOverlayViewWillActivate(_ coachingOverlayView: ARCoachingOverlayView) {
    upperControlsView.isHidden = true
}

When the coaching overlay determines that the goal has been met, it disappears from the user’s view. ARKit notifies your delegate that the coaching process has ended, which is when you show your app’s main user interface.

func coachingOverlayViewDidDeactivate(_ coachingOverlayView: ARCoachingOverlayView) {
    upperControlsView.isHidden = false
}

Place Virtual Content

To give the user an idea of where they can place virtual content, annotate the environment to give them a preview. The sample app draws a square that gives the user visual confirmation of the shape and alignment of the surfaces that ARKit is aware of.

To figure out where to put the square in the real world, you use an ARRaycastQuery to ask ARKit where any surfaces exist in the real world. First, you create a ray-cast query that defines the 2D point on the screen you’re interested in. Because the focus square is aligned with the center of the screen, you create a query for the screen center.

func getRaycastQuery() -> ARRaycastQuery? {
    return sceneView.raycastQuery(from: screenCenter, allowing: .estimatedPlane, alignment: .any)
}

Then, you execute the ray-cast query by asking the session to cast it.

func getTrackedRaycast(_ query: ARRaycastQuery, virtualObject: VirtualObject) -> ARTrackedRaycast? {
    return session.trackedRaycast(query) { (results) in
        self.setVirtualObject3DPosition(results, with: virtualObject)
    }
}

ARKit returns a position in the results parameter that includes the depth of where that point lies on a surface in the real world, and how the surface is angled with respect to gravity. That information is wrapped in the ray cast result’s worldTransform, which you set on your virtual object’s simdWorldTransform to effectively place your virtual object in the physical environment.

func setPosition(of virtualObject: VirtualObject, with result: ARRaycastResult) {
    virtualObject.simdWorldTransform = result.worldTransform
}

If your app offers different types of virtual content, give the user an interface to choose from. The sample app exposes a selection menu when the user taps the plus button. When the user chooses an item from the list, you instantiate the corresponding 3D model and anchor it in the world at the focus square’s current position.

func placeVirtualObject(_ virtualObject: VirtualObject) {
    guard focusSquare.state != .initializing,
    let query = sceneView.raycastQuery(from: screenCenter, allowing: .estimatedPlane, alignment: virtualObject.allowedAlignment) else {
        self.statusViewController.showMessage("CANNOT PLACE OBJECT\nTry moving left or right.")
        if let controller = self.objectsViewController {
            self.virtualObjectSelectionViewController(controller, didDeselectObject: virtualObject)
        }
        return
    }
    
    virtualObject.raycast = getTrackedRaycast(query, virtualObject: virtualObject)
    virtualObjectInteraction.selectedObject = virtualObject
}

Refine the Position of Virtual Content Over Time

As the session runs, ARKit analyzes each camera image and learns more about the layout of the physical environment. When ARKit updates its estimated size, shape, and position of real-world planes, you may need to update the position of your app’s virtual content. Opt in to notification of these events by creating an ARTrackedRaycast, and providing a ray-cast query that tells ARKit which updates you’re interested in.

func getTrackedRaycast(_ query: ARRaycastQuery, virtualObject: VirtualObject) -> ARTrackedRaycast? {
    return session.trackedRaycast(query) { (results) in
        self.setVirtualObject3DPosition(results, with: virtualObject)
    }
}

As ARKit repeats your ray-cast query, it gives you the results only when they differ from prior results. Pass a closure to the trackedRaycast(_:updateHandler:) to define your response to these events, where you update the position of your app’s virtual content accordingly.

func setVirtualObject3DPosition(_ results: [ARRaycastResult], with virtualObject: VirtualObject) {
    guard let result = results.first else {
        fatalError("Unexpected case: the update handler is always supposed to return at least one result.")
    }
    
    if virtualObject.allowedAlignment == .any && self.virtualObjectInteraction.trackedObject == virtualObject {
        
        // If an object that's aligned to a surface is being dragged, then
        // smoothen its orientation to avoid visible jumps, and apply only the translation directly.
        virtualObject.simdWorldPosition = result.worldTransform.translation
        
        let previousOrientation = virtualObject.simdWorldTransform.orientation
        let currentOrientation = result.worldTransform.orientation
        virtualObject.simdWorldOrientation = simd_slerp(previousOrientation, currentOrientation, 0.1)
    } else {
        self.setPosition(of: virtualObject, with: result)
    }
    
    // If the virtual object is not yet in the scene, add it.
    if virtualObject.parent == nil {
        self.sceneView.scene.rootNode.addChildNode(virtualObject)
        virtualObject.shouldUpdateAnchor = true
    }
    
    if virtualObject.shouldUpdateAnchor {
        virtualObject.shouldUpdateAnchor = false
        self.updateQueue.async {
            self.sceneView.addOrUpdateAnchor(for: virtualObject)
        }
    }
}

Manage Tracked Ray Casts

Because ARKit continues to call them, tracked ray casts can increasingly consume resources with the more virtual content a user places. Stop the tracked ray cast when you’re satisfied with its associated virtual object’s position, or when you remove the virtual object from your scene.

func removeVirtualObject(at index: Int) {
    guard loadedObjects.indices.contains(index) else { return }
    
    loadedObjects[index].removeFromParentNode()
    loadedObjects[index].raycast?.stopTracking()
    loadedObjects[index].raycast = nil
    loadedObjects[index].unload()
    loadedObjects.remove(at: index)
}

Enable User Interaction with Virtual Content

To allow users to move virtual content in the world after they’ve placed it, implement a pan gesture recognizer.

func createPanGestureRecognizer(_ sceneView: VirtualObjectARView) {
    let panGesture = ThresholdPanGesture(target: self, action: #selector(didPan(_:)))
    panGesture.delegate = self
    sceneView.addGestureRecognizer(panGesture)
}

When you move a virtual object, you keep the object subscribed for positional updates from ARKit by updating its tracked ray cast with a new query.

func translate(_ object: VirtualObject, basedOn screenPos: CGPoint) {
    if let query = sceneView.raycastQuery(from: screenPos, allowing: .estimatedPlane, alignment: object.allowedAlignment) {
        object.raycast?.update(query)
    }
}

Ray casting gives you orientation information about the surface at a given screen point. While dragging, you avoid quick changes in orientation by subtracting the gesture’s rotation from the current object rotation.

@objc
func didRotate(_ gesture: UIRotationGestureRecognizer) {
    guard gesture.state == .changed else { return }
    
    trackedObject?.objectRotation -= Float(gesture.rotation)
    
    gesture.rotation = 0
}

Handle Interruption in Tracking

In cases where tracking conditions are poor, ARKit invokes your delegate’s sessionWasInterrupted(_:). In these circumstances, the positions of your app’s virtual content may be inaccurate with respect to the camera feed, so you hide your virtual content.

func hideVirtualContent() {
    virtualObjectLoader.loadedObjects.forEach { $0.isHidden = true }
}

Restore your app’s virtual content when tracking conditions improve. To notify you of improved conditions, ARKit calls your delegate’s session(_:cameraDidChangeTrackingState:) function, passing in a camera trackingState equal to ARTrackingStateNormal.

func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera) {
    statusViewController.showTrackingQualityInfo(for: camera.trackingState, autoHide: true)
    switch camera.trackingState {
    case .notAvailable, .limited:
        statusViewController.escalateFeedback(for: camera.trackingState, inSeconds: 3.0)
    case .normal:
        statusViewController.cancelScheduledMessage(for: .trackingStateEscalation)
        showVirtualContent()
    }
}

Restore an Interrupted AR Experience

When a session is interrupted, ARKit asks if you want to try to restore the AR experience. You do that by opting in to relocalization, by overriding sessionShouldAttemptRelocalization(_:) and returning true.

func sessionShouldAttemptRelocalization(_ session: ARSession) -> Bool {
    return true
}

During relocalization, the coaching overlay displays tailored instructions to the user. To allow the user to focus on the coaching process, hide your app’s UI when coaching is enabled.

func coachingOverlayViewWillActivate(_ coachingOverlayView: ARCoachingOverlayView) {
    upperControlsView.isHidden = true
}

When ARKit succeeds in restoring the experience, show your app’s UI again so everything appears the way it was before the interruption. When the coaching overlay disappears from the user’s view, ARKit invokes your coachingOverlayViewDidDeactivate(_:) callback, which is where you restore your app’s UI.

func coachingOverlayViewDidDeactivate(_ coachingOverlayView: ARCoachingOverlayView) {
    upperControlsView.isHidden = false
}

Enable the User to Start Over Rather Than Restore

If the user decides to give up on restoring the session, you restart the experience in your delegate’s coachingOverlayViewDidRequestSessionReset(_:) function. ARKit invokes this callback when the user taps the coaching overlay’s Start Over button.

func coachingOverlayViewDidRequestSessionReset(_ coachingOverlayView: ARCoachingOverlayView) {
    restartExperience()
}

See Also

World Tracking

Understanding World Tracking

Discover supporting concepts, features, and best practices for building great AR experiences.

class ARWorldTrackingConfiguration

A configuration that monitors the iOS device's position and orientation, while enabling you to augment the environment that's in front of the user.

class ARPlaneAnchor

A 2D surface that ARKit detects in the physical environment.

Tracking and Visualizing Planes

Detect surfaces in the physical environment and visualize their shape and location in 3D space.

class ARCoachingOverlayView

A view that presents visual instructions that guide the user during session initialization and in limited tracking situations.

Beta
class ARWorldMap

The space-mapping state and set of anchors from a world-tracking AR session.

Saving and Loading World Data

Serialize a world tracking session to resume it later on.

Ray-Casting and Hit-Testing

Find 3D positions on real-world surfaces given a screen point.

Beta Software

This documentation contains preliminary information about an API or technology in development. This information is subject to change, and software implemented according to this documentation should be tested with final operating system software.

Learn more about using Apple's beta software