- iOS 13.0+
- Xcode 11.0+
When tracking users’ faces in a world-tracking session, ARKit incorporates information from both the front and rear camera feeds in the AR experience. In addition to tracking the physical environment using the rear camera, ARKit uses the front camera to deliver an anchor that provides the position and expression of the user’s face.
To demonstrate applying the user’s face while world-tracking, this sample app lets the user place robot heads that reflect the user’s facial expression.
Configure and Start the Session
This app tracks the user’s face in a world-tracking session on iOS 13 and iPad OS 13 or later, on devices with a front TrueDepth camera. To prevent the app from running an unsupported configuration, check whether the iOS device supports simultaneous world and user face-tracking.
If the device running the app doesn’t support user face-tracking in a world-tracking session, the sample project will stop. In your app, consider gracefully degrading the AR experience in this case, such as by presenting the user with an error message and continuing the experience without it.
The sample app sets the
user property to
true on the world-tracking configuration when app loads the view controller.
The sample app then starts the session by running the configuration when the view controller is about to appear onscreen.
Preview Virtual Content in the Physical Environment
The app checks whether a robot head preview exists and creates one if not. ARKit calls the implementation of
session(_: every frame, which makes it a good location for a periodic check.
Detect Changes in the User’s Expression
ARKit provides the app with an updated anchor when the user changes their expression, position, or orientation with respect to the world. If there’s an active robot head preview, the app applies these changes to the head.
Inspect Expression Information
In the robot head’s
update(with face function, the app reads the user’s current expression by interpreting the anchor’s blend shapes.
Blend shapes are
Float values normalized within the range
0 representing the facial feature’s rest position, and
1 representing the opposite––the feature in its most pronounced state. To begin processing the values, the app stores them locally by accessing the anchor’s
React to the User’s Expression
Blend shape values can apply in unique ways depending on an app’s requirements. The sample app uses blend shapes to make the robot head appear to mimic the user’s expression, such as applying the brow and lip values to offset the robot’s brow and lip positions.
The entity for the robot’s eye opens or closes when the sample app applies the corresponding blend shape value as a scale factor.
Position the Robot Head
In addition to capturing the user’s expression using the front camera, ARKit records the position of the user’s face with respect to the world. By design, the user’s face anchor is always located behind the rear camera. To serve the goal of mimicking the user with the robot head, the sample app applies the face anchor’s position to make the robot head always visible. First, it sets the robot head’s initial position equal to that of the camera.
Then the app offsets its z-position in the same amount as the camera’s distance from the user’s face.
Orient the Robot Head
The sample app also uses the anchor’s orientation to direct the front of the robot’s head continually toward the camera. It starts by accessing the anchor’s orientation.
Then it adds
pi to the
y-Euler angle to turn it on the y-axis.
To effect the change, the app applies the updated Euler angles to the robot head’s orientation.
Capture the Expression by Placing the Head
To demonstrate the variety of expressions tracked during the session, the sample app places the robot head in the physical environment when the user taps the screen. When the app initially previews the expressions, it positions the robot head at a fixed offset from the camera. When the user taps the screen, the app reanchors the robot head by updating its position to its current world location.
nil prevents the app from updating the facial expression in
session(did, which freezes that expression on the placed robot head.
When ARKit calls
session(did again, the app checks whether a robot head preview exists, and creates one if not.
When the app sets
nil, it creates another robot head preview, continuing the user’s ability to place objects and archive additional facial expressions.