Configuring Camera Capture to Collect a Portrait Effects Matte

Prepare your app to capture a portrait effects matte when taking photos.


When capturing a photo in the iOS Camera app, you can cause the portrait effects matte to be generated by entering Portrait mode and detecting a face. You can tell whether the camera found a face by checking for four yellow brackets in the viewfinder; photographing a detected face in Portrait mode doesn't guarantee the generation of a matte, but it is a prerequisite.

Like the iOS Camera app, your app can also opt in for creating and embedding portrait effects matte into captured photos, eventually saving them as HEIC and JPEG images. Using AVFoundation’s AVCapturePhotoOutput, your app can find out whether a particular camera configuration supports the delivery of a portrait effect mattes to still images, and opt in to have them delivered on a per-photo request basis. AVCapturePhotoOutput delivers photo results using an in-memory wrapper object called AVCapturePhoto.

// private let photoOutput = AVCapturePhotoOutput()
// Check that portrait effects matte delivery is supported on this particular device:
if self.photoOutput.isPortraitEffectsMatteDeliverySupported {
    self.photoOutput.isPortraitEffectsMatteDeliveryEnabled = true

If you’ve opted in for portrait effects matte delivery, you can inspect the AVCaptureResolvedPhotoSettings passed to you in your AVCapturePhotoCaptureDelegate, and read the portraitEffectsMatteDimensions property to see what size matting image will be delivered, instead of hard-coding assumptions about the size.

Even if your app opts in to matte delivery, you're not guaranteed delivery of a matting image in the resulting AVCapturePhoto. The neural network responsible for generating portrait effects mattes is trained to detect and segment people. If there are no people in the image scene, the image won't benefit from a portrait effects matte, so none is delivered. The AVCapturePhoto object has a new read-only accessor property in iOS 12 for accessing the image’s portraitEffectsMatte. If you read this property and its return value is nil, no portrait effects matte was generated for the image (because none was needed). The camera generates the matte image as a post-processing step, so you can't check whether a matte will be generated before capturing the photo. The best you can do is check for detected faces by using AVMetadataFaceObject and enabling depth and portrait effects matte.

The portrait effects matte algorithm uses a combination of person segmentation and depth as a refinement, so matte delivery is only supported for cameras and camera formats that support depth. Portrait effects matte images shouldn't be embedded in image files that contain no depth image; the two should travel together. This rule is enforced at the AVCapturePhotoOutput level. If you opt in for portrait effects matte delivery and ask for it to be embedded in the AVCapturePhotoOutput container, you must also opt in for embedded depth data delivery.

Write a Captured Portrait Effects Matte to Disk

AVCapturePhoto offers clients access to the underlying in-memory file container using fileDataRepresentation(). The resulting NSData is suitable for writing to disk as a HEIC or JPEG file. To replace or strip the portrait effects matte in the image file, call fileDataRepresentation(with:), which invokes a customizer object to modify, replace, or strip various pieces of the image before packaging it into a container suitable for writing to disk.

Both JPEG and HEIC files can store embedded portrait effects matte data. When stored in JFIF/JPEG, the matte is compressed using JPEG. When stored in HEIC files, it's compressed using the HEVC codec. The matting images tend to contain large areas of flat black or white, so they compress to a file size smaller than iOS 11 photos.

See Also

Creating a Portrait Effects Matte

init(fromDictionaryRepresentation: [AnyHashable : Any])

Initializes a portrait effects matte instance from auxiliary image information in an image file.

func applyingExifOrientation(CGImagePropertyOrientation) -> Self

Returns a derivative portrait effects matte after applying the specified EXIF orientation.

func replacingPortraitEffectsMatte(with: CVPixelBuffer) -> Self

Returns a portrait effects matte by wrapping the replacement pixel buffer.