The TrueDepth camera provides depth data in real time that allows you to determine the distance of a pixel from the front-facing camera. This sample demonstrates how to use the AVFoundation framework’s capture API to read data from the TrueDepth camera, and how to display it in an intuitive fashion onscreen.
The sample shows two different views: a 2D view that distinguishes depth values by mapping depth to color, and a 3D view that renders data as a point cloud.
To see this sample app in action, build and run the project in Xcode on an iOS device running iOS 11 or later. Because Xcode doesn’t have access to the TrueDepth camera, this sample will not build or run in the Xcode simulator.
Set Up a Capture Session
Set up an AVCaptureSession on a separate thread via the session queue. Initialize this session queue before configuring the camera for capture, like so:
The startRunning method is a blocking call that may take time to execute. Dispatch session setup to the session queue so the main queue isn’t blocked, allowing the app’s UI to stay responsive:
Setting up the camera for video capture follows many of the same steps as normal video capture. See Setting Up a Capture Session for details on configuring streaming setup.
On top of normal setup, request depth data by declaring a separate output:
Explicitly add this output type to your capture session:
Search for the highest resolution available with floating-point depth values, and lock the configuration to the format.
Synchronize the normal RGB video data with depth data output. The first output in the dataOutputs array is the master output.
The CameraViewController implementation creates and manages this session to interface with the camera. It also contains UI to toggle between the two viewing modes, 2D and 3D.
Visualize Depth Data in 2D
The sample uses JET color coding to distinguish depth values, ranging from red (close) to blue (far). A slider controls the blending of the color code and the actual color values. Touching a pixel displays its depth value.
DepthToJETConverter performs the conversion. It separates the color spectrum into histogram bins, colors a Metal texture from depth values obtained in the image buffer, and renders that texture into the preview.
Visualize Depth Data in 3D
The sample’s 3D viewer renders data as a point cloud. Control the camera with the following gestures:
Pinch to zoom.
Pan to move the camera around the center.
Rotate with two fingers to turn the camera angle.
Double-tap the screen to reset the initial position.
The sample implements a 3D point cloud as a PointCloudMetalView. It uses a Metal vertex shader to control geometry and a Metal fragment shader to color individual vertices, keeping the depth texture and color texture separate:
The depth frame’s depth map provides the basis for the Metal view’s depth texture:
The RGB image provides the basis for the Metal view’s color texture:
Track Thermal State
Processing depth data from a live stream may cause the device to heat up. Keep tabs on the thermal state so you can alert the user if it exceeds a dangerous threshold.
An object that wraps a matting image for a particular semantic segmentation.
This documentation contains preliminary information about an API or technology in development. This information is subject to change, and software implemented according to this documentation should be tested with final operating system software.