Media Capture

Configuring a Session

You use a preset on the session to specify the image quality and resolution you want. A preset is a constant that identifies one of a number of possible configurations; in some cases the actual configuration is device-specific:

For examples of the actual values these presets represent for various devices, see “Saving to a Movie File” and “Capturing Still Images.”

If you want to set a size-specific configuration, you should check whether it is supported before setting it:

if ([session canSetSessionPreset:AVCaptureSessionPreset1280x720]) {

    session.sessionPreset = AVCaptureSessionPreset1280x720;

}

else {

    // Handle the failure.

}

In many situations, you create a session and the various inputs and outputs all at once. Sometimes, however, you may want to reconfigure a running session, perhaps as different input devices become available, or in response to user request. This can present a challenge, since, if you change them one at a time, a new setting may be incompatible with an existing setting. To deal with this, you use beginConfiguration and commitConfiguration to batch multiple configuration operations into an atomic update. After calling beginConfiguration, you can for example add or remove outputs, alter the sessionPreset, or configure individual capture input or output properties. No changes are actually made until you invoke commitConfiguration, at which time they are applied together.

[session beginConfiguration];

// Remove an existing capture device.

// Add a new capture device.

// Reset the preset.

[session commitConfiguration];

Monitoring Capture Session State

A capture session posts notifications that you can observe to be notified, for example, when it starts or stops running, or when it is interrupted. You can also register to receive an AVCaptureSessionRuntimeErrorNotification if a runtime error occurs. You can also interrogate the session’s running property to find out if it is running, and its interrupted property to find out if it is interrupted.

Device Characteristics

You can ask a device about several different characteristics. You can test whether it provides a particular media type or supports a given capture session preset using hasMediaType: and supportsAVCaptureSessionPreset: respectively. To provide information to the user, you can find out the position of the capture device (whether it is on the front or the back of the unit they’re using), and its localized name. This may be useful if you want to present a list of capture devices to allow the user to choose one.

The following code example iterates over all the available devices and logs their name, and for video devices their position on the unit.

NSArray *devices = [AVCaptureDevice devices];

for (AVCaptureDevice *device in devices) {

    NSLog(@"Device name: %@", [device localizedName]);

    if ([device hasMediaType:AVMediaTypeVideo]) {

        if ([device position] == AVCaptureDevicePositionBack) {

            NSLog(@"Device position : back");

        }

        else {

            NSLog(@"Device position : front");

        }

    }

}

In addition, you can find out the device’s model ID and its unique ID.

Device Capture Settings

Different devices have different capabilities; for example, some may support different focus or flash modes; some may support focus on a point of interest.

The following code fragment shows how you can find video input devices that have a torch mode and support a given capture session preset:

NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];

NSMutableArray *torchDevices = [[NSMutableArray alloc] init];

for (AVCaptureDevice *device in devices) {

    [if ([device hasTorch] &&

         [device supportsAVCaptureSessionPreset:AVCaptureSessionPreset640x480]) {

        [torchDevices addObject:device];

    }

}

If you find multiple devices that meet your criteria, you might let the user choose which one they want to use. To display a description of a device to the user, you can use its localizedName property.

You use the various different features in similar ways. There are constants to specify a particular mode, and you can ask a device whether it supports a particular mode. In several cases you can observe a property to be notified when a feature is changing. In all cases, you should lock the device before changing the mode of a particular feature, as described in “Configuring a Device.”

if ([currentDevice isFocusModeSupported:AVCaptureFocusModeContinuousAutoFocus]) {

    CGPoint autofocusPoint = CGPointMake(0.5f, 0.5f);

    [currentDevice setFocusPointOfInterest:autofocusPoint];

    [currentDevice setFocusMode:AVCaptureFocusModeContinuousAutoFocus];

}

if ([currentDevice isExposureModeSupported:AVCaptureExposureModeContinuousAutoExposure]) {

    CGPoint exposurePoint = CGPointMake(0.5f, 0.5f);

    [currentDevice setExposurePointOfInterest:exposurePoint];

    [currentDevice setExposureMode:AVCaptureExposureModeContinuousAutoExposure];

}

Configuring a Device

To set capture properties on a device, you must first acquire a lock on the device using lockForConfiguration:. This avoids making changes that may be incompatible with settings in other applications. The following code fragment illustrates how to approach changing the focus mode on a device by first determining whether the mode is supported, then attempting to lock the device for reconfiguration. The focus mode is changed only if the lock is obtained, and the lock is released immediately afterward.

if ([device isFocusModeSupported:AVCaptureFocusModeLocked]) {

    NSError *error = nil;

    if ([device lockForConfiguration:&error]) {

        device.focusMode = AVCaptureFocusModeLocked;

        [device unlockForConfiguration];

    }

    else {

        // Respond to the failure as appropriate.

You should only hold the device lock if you need settable device properties to remain unchanged. Holding the device lock unnecessarily may degrade capture quality in other applications sharing the device.

Switching Between Devices

Sometimes you may want to allow the user to switch between input devices—for example, on an iPhone 4 they could switch from using the front to the back camera. To avoid pauses or stuttering, you can reconfigure a session while it is running, however you should use beginConfiguration and commitConfiguration to bracket your configuration changes:

AVCaptureSession *session = <#A capture session#>;

[session beginConfiguration];

[session removeInput:frontFacingCameraDeviceInput];

[session addInput:backFacingCameraDeviceInput];

[session commitConfiguration];

When the outermost commitConfiguration is invoked, all the changes are made together. This ensures a smooth transition.

Saving to a Movie File

You save movie data to a file using an AVCaptureMovieFileOutput object. (AVCaptureMovieFileOutput is a concrete subclass of AVCaptureFileOutput, which defines much of the basic behavior.) You can configure various aspects of the movie file output, such as the maximum duration of the recording, or the maximum file size. You can also prohibit recording if there is less than a given amount of disk space left.

AVCaptureMovieFileOutput *aMovieFileOutput = [[AVCaptureMovieFileOutput alloc] init];

CMTime maxDuration = <#Create a CMTime to represent the maximum duration#>;

aMovieFileOutput.maxRecordedDuration = maxDuration;

aMovieFileOutput.minFreeDiskSpaceLimit = <#An appropriate minimum given the quality of the movie format and the duration#>;

The resolution and bit rate for the output depend on the capture session’s sessionPreset. The video encoding is typically H.264 and audio encoding AAC. The actual values vary by device, as illustrated in the following table.

AVCaptureMovieFileOutput *aMovieFileOutput = <#Get a movie file output#>;

NSURL *fileURL = <#A file URL that identifies the output location#>;

[aMovieFileOutput startRecordingToOutputFileURL:fileURL recordingDelegate:<#The delegate#>];

- (void)captureOutput:(AVCaptureFileOutput *)captureOutput

        didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL

        fromConnections:(NSArray *)connections

        error:(NSError *)error {

    BOOL recordedSuccessfully = YES;

    if ([error code] != noErr) {

        // A problem occurred: Find out if the recording was successful.

        id value = [[error userInfo] objectForKey:AVErrorRecordingSuccessfullyFinishedKey];

        if (value) {

            recordedSuccessfully = [value boolValue];

        }

    }

    // Continue as appropriate...

AVCaptureMovieFileOutput *aMovieFileOutput = <#Get a movie file output#>;

NSArray *existingMetadataArray = aMovieFileOutput.metadata;

NSMutableArray *newMetadataArray = nil;

if (existingMetadataArray) {

    newMetadataArray = [existingMetadataArray mutableCopy];

}

else {

    newMetadataArray = [[NSMutableArray alloc] init];

}

AVMutableMetadataItem *item = [[AVMutableMetadataItem alloc] init];

item.keySpace = AVMetadataKeySpaceCommon;

item.key = AVMetadataCommonKeyLocation;

CLLocation *location - <#The location to set#>;

item.value = [NSString stringWithFormat:@"%+08.4lf%+09.4lf/"

    location.coordinate.latitude, location.coordinate.longitude];

[newMetadataArray addObject:item];

aMovieFileOutput.metadata = newMetadataArray;

Processing Frames of Video

An AVCaptureVideoDataOutput object uses delegation to vend video frames. You set the delegate using setSampleBufferDelegate:queue:. In addition to the delegate, you specify a serial queue on which they delegate methods are invoked. You must use a serial queue to ensure that frames are delivered to the delegate in the proper order. You should not pass the queue returned by dispatch_get_current_queue since there is no guarantee as to which thread the current queue is running on. You can use the queue to modify the priority given to delivering and processing the video frames.

The frames are presented in the delegate method, captureOutput:didOutputSampleBuffer:fromConnection:, as instances of the CMSampleBuffer opaque type (see “Representations of Media”). By default, the buffers are emitted in the camera’s most efficient format. You can use the videoSettings property to specify a custom output format. The video settings property is a dictionary; currently, the only supported key is kCVPixelBufferPixelFormatTypeKey. The recommended pixel format choices for iPhone 4 are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange or kCVPixelFormatType_32BGRA; for iPhone 3G the recommended pixel format choices are kCVPixelFormatType_422YpCbCr8 or kCVPixelFormatType_32BGRA. You should use the CVOpenGLESTextureCacheRef type defined in the Core Video framework if you want to work with the 420v format directly. Both Core Graphics and OpenGL work well with the BGRA format:

AVCaptureSession *captureSession = <#Get a capture session#>;

NSDictionary *newSettings =

                @{ (NSString *)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };

captureSession.videoSettings = newSettings;

Capturing Still Images

You use an AVCaptureStillImageOutput output if you want to capture still images with accompanying metadata. The resolution of the image depends on the preset for the session, as illustrated in this table:

AVCaptureStillImageOutput *stillImageOutput = [[AVCaptureStillImageOutput alloc] init];

NSDictionary *outputSettings = @{ AVVideoCodecKey : AVVideoCodecJPEG};

[stillImageOutput setOutputSettings:outputSettings];

AVCaptureConnection *videoConnection = nil;

for (AVCaptureConnection *connection in stillImageOutput.connections) {

    for (AVCaptureInputPort *port in [connection inputPorts]) {

        if ([[port mediaType] isEqual:AVMediaTypeVideo] ) {

            videoConnection = connection;

            break;

        }

    }

    if (videoConnection) { break; }

}

[stillImageOutput captureStillImageAsynchronouslyFromConnection:videoConnection completionHandler:

    ^(CMSampleBufferRef imageSampleBuffer, NSError *error) {

        CFDictionaryRef exifAttachments =

            CMGetAttachment(imageSampleBuffer, kCGImagePropertyExifDictionary, NULL);

        if (exifAttachments) {

            // Do something with the attachments.

        }

        // Continue as appropriate.

    }];

Video Preview

You can provide the user with a preview of what’s being recorded using an AVCaptureVideoPreviewLayer object. AVCaptureVideoPreviewLayer is a subclass ofCALayer (see Core Animation Programming Guide. You don’t need any outputs to show the preview.

Unlike a capture output, a video preview layer maintains a strong reference to the session with which it is associated. This is to ensure that the session is not deallocated while the layer is attempting to display video. This is reflected in the way you initialize a preview layer:

AVCaptureSession *captureSession = <#Get a capture session#>;

CALayer *viewLayer = <#Get a layer from the view in which you want to present the preview#>;

AVCaptureVideoPreviewLayer *captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:captureSession];

[viewLayer addSublayer:captureVideoPreviewLayer];

In general, the preview layer behaves like any other CALayer object in the render tree (see Core Animation Programming Guide). You can scale the image and perform transformations, rotations and so on just as you would any layer. One difference is that you may need to set the layer’s orientation property to specify how it should rotate images coming from the camera. In addition, on iPhone 4 the preview layer supports mirroring (this is the default when previewing the front-facing camera).

Showing Audio Levels

To monitor the average and peak power levels in an audio channel in a capture connection, you use an AVCaptureAudioChannel object. Audio levels are not key-value observable, so you must poll for updated levels as often as you want to update your user interface (for example, 10 times a second).

AVCaptureAudioDataOutput *audioDataOutput = <#Get the audio data output#>;

NSArray *connections = audioDataOutput.connections;

if ([connections count] > 0) {

    // There should be only one connection to an AVCaptureAudioDataOutput.

    AVCaptureConnection *connection = [connections objectAtIndex:0];

    NSArray *audioChannels = connection.audioChannels;

    for (AVCaptureAudioChannel *channel in audioChannels) {

        float avg = channel.averagePowerLevel;

        float peak = channel.peakHoldLevel;

        // Update the level meter user interface.

    }

}

Create and Configure a Capture Session

You use an AVCaptureSession object to coordinate the flow of data from an AV input device to an output. Create a session, and configure it to produce medium resolution video frames.

AVCaptureSession *session = [[AVCaptureSession alloc] init];

session.sessionPreset = AVCaptureSessionPresetMedium;

Create and Configure the Device and Device Input

Capture devices are represented by AVCaptureDevice objects; the class provides methods to retrieve an object for the input type you want. A device has one or more ports, configured using an AVCaptureInput object. Typically, you use the capture input in its default configuration.

Find a video capture device, then create a device input with the device and add it to the session.

AVCaptureDevice *device =

        [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

NSError *error = nil;

AVCaptureDeviceInput *input =

        [AVCaptureDeviceInput deviceInputWithDevice:device error:&error];

if (!input) {

    // Handle the error appropriately.

}

[session addInput:input];

Create and Configure the Data Output

You use an AVCaptureVideoDataOutput object to process uncompressed frames from the video being captured. You typically configure several aspects of an output. For video, for example, you can specify the pixel format using the videoSettings property, and cap the frame rate by setting the minFrameDuration property.

Create and configure an output for video data and add it to the session; cap the frame rate to 15 fps by setting the minFrameDuration property to 1/15 second:

AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];

[session addOutput:output];

output.videoSettings =

                @{ (NSString *)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };

output.minFrameDuration = CMTimeMake(1, 15);

The data output object uses delegation to vend the video frames. The delegate must adopt the AVCaptureVideoDataOutputSampleBufferDelegate protocol. When you set the data output’s delegate, you must also provide a queue on which callbacks should be invoked.

dispatch_queue_t queue = dispatch_queue_create("MyQueue", NULL);

[output setSampleBufferDelegate:self queue:queue];

dispatch_release(queue);

You use the queue to modify the priority given to delivering and processing the video frames.

Implement the Sample Buffer Delegate Method

In the delegate class, implement the method (captureOutput:didOutputSampleBuffer:fromConnection:) that is called when a sample buffer is written. The video data output object delivers frames as CMSampleBuffers, so you need to convert from the CMSampleBuffer to a UIImage object. The function for this operation is shown in “Converting a CMSampleBuffer to a UIImage.”

- (void)captureOutput:(AVCaptureOutput *)captureOutput

         didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer

         fromConnection:(AVCaptureConnection *)connection {

    UIImage *image = imageFromSampleBuffer(sampleBuffer);

    // Add your code here that uses the image.

}

Remember that the delegate method is invoked on the queue you specified in setSampleBufferDelegate:queue:; if you want to update the user interface, you must invoke any relevant code on the main thread.

Starting and Stopping Recording

After configuring the capture session, you send it a startRunning message to start the recording.

[session startRunning];

To stop recording, you send the session a stopRunning message.