Prepare your app to capture a portrait effects matte when taking photos.
When capturing a photo in the iOS Camera app, you can cause the portrait effects matte to be generated by entering Portrait mode and detecting a face. You can tell whether the camera found a face by checking for four yellow brackets in the viewfinder; photographing a detected face in Portrait mode doesn't guarantee the generation of a matte, but it is a prerequisite.
Like the iOS Camera app, your app can also opt in for creating and embedding portrait effects matte into captured photos, eventually saving them as HEIC and JPEG images. Using AVFoundation’s
AVCapture, your app can find out whether a particular camera configuration supports the delivery of a portrait effect mattes to still images, and opt in to have them delivered on a per-photo request basis.
AVCapture delivers photo results using an in-memory wrapper object called
If you’ve opted in for portrait effects matte delivery, you can inspect the
AVCapture passed to you in your
AVCapture, and read the
portrait property to see what size matting image will be delivered, instead of hard-coding assumptions about the size.
Even if your app opts in to matte delivery, you're not guaranteed delivery of a matting image in the resulting
AVCapture. The neural network responsible for generating portrait effects mattes is trained to detect and segment people. If there are no people in the image scene, the image won't benefit from a portrait effects matte, so none is delivered. The
AVCapture object has a new read-only accessor property in iOS 12 for accessing the image’s
portrait. If you read this property and its return value is
nil, no portrait effects matte was generated for the image (because none was needed). The camera generates the matte image as a post-processing step, so you can't check whether a matte will be generated before capturing the photo. The best you can do is check for detected faces by using
AVMetadata and enabling depth and portrait effects matte.
The portrait effects matte algorithm uses a combination of person segmentation and depth as a refinement, so matte delivery is only supported for cameras and camera formats that support depth. Portrait effects matte images shouldn't be embedded in image files that contain no depth image; the two should travel together. This rule is enforced at the
AVCapture level. If you opt in for portrait effects matte delivery and ask for it to be embedded in the
AVCapture container, you must also opt in for embedded depth data delivery.
Write a Captured Portrait Effects Matte to Disk
AVCapture offers clients access to the underlying in-memory file container using
file. The resulting
NSData is suitable for writing to disk as a HEIC or JPEG file. To replace or strip the portrait effects matte in the image file, call
file, which invokes a customizer object to modify, replace, or strip various pieces of the image before packaging it into a container suitable for writing to disk.
Both JPEG and HEIC files can store embedded portrait effects matte data. When stored in JFIF/JPEG, the matte is compressed using JPEG. When stored in HEIC files, it's compressed using the HEVC codec. The matting images tend to contain large areas of flat black or white, so they compress to a file size smaller than iOS 11 photos.