Real-Time Media Capture

The QTKit capture API is optimized to provide capturing real-time media from devices such as iSight cameras, camcorders and tape decks, microphones and audio interfaces. The captured media can then be outputted to QuickTime movies, audio and video previews, as well as raw decompressed frames for further processing.

Capture Capabilities

The capture capability provided by QTKit capture API includes frame-accurate, audio/video synchronization, in addition to enabling you to preview captured content and save it to a file. Frame-accurate capture means you can specify that capture and recording of media will occur with specific timecodes and other per-sample metadata. For example, using the QTKit capture APIs, you can specify precisely the number of frames in a file that you want to write out.

The QTKit capture API also provides access to transport controls, such as fast-forward and rewind, on a video camcorder. In addition, you can write captured QuickTime data to a file while providing an onscreen preview.

In addition, the QTKit API provides you with the ability to share the iSight and other webcams between multiple applications.

Core Classes

There are four classes in this group that can best be described as core classes, as shown in Table 2-1. Understanding how these classes work and which methods to use in your application development is essential to taking full advantage of the QTKit capture API.

Of the classes in this group, the most useful is QTCaptureSession. This class provides an interface for connecting input sources to output destinations. The method used most commonly in this class is startRunning, which tells the receiver to start capturing data from its inputs and then to send that data to its outputs. Notably, if you’re using this method, when data does not need to be sent to file outputs, previews, or other outputs, your capture session should not be running, so that the overhead from capturing does not affect the performance of your application.

The other classes in this group, QTCaptureInput and QTCaptureOutput, which are both abstract classes, provide interfaces for connecting inputs and outputs. An input source can have multiple connections, which is common for many cameras that have both audio and video output streams. When you use QTCaptureOutput objects, you don’t need to have a fixed number of connections, but you do need a destination for your capture session and all of its input data.

The last of the four classes in this group, QTCaptureConnection, represents a connection over which a single stream of media data is sent from a QTCaptureInput to a QTCaptureSession and from a QTCaptureSession to a QTCaptureOutput.

Input/Output Classes

There are six concrete output classes and one input class belonging to this group, as shown in Table 2-2.

You can use the methods available in the QTCaptureDeviceInput class, for example, to handle input sources for various media devices, such as cameras and microphones. The six output classes provide output destinations for QTCaptureSession objects that can be used to write captured media to QuickTime movies or to preview video or audio that is being captured. QTCaptureFileOutput, an abstract superclass, provides an output destination for a capture session to write captured media simply to files.

Note that the delegate methods shown in Table 2-2 belong to NSObject and cannot be called on these classes.

Utility Classes

There are three classes belonging to this group: QTCompressionOptions, QTFormatDescription, and QTSampleBuffer, shown in Table 2-3. These are best characterized as utility classes, in that they perform tasks related to representing, for example, the compressions for particular media, or describing the formats of various media samples.

Of the classes in this group, one of the most useful is QTCompressionOptions, which lets you describe compression options for all kinds of different media, using the compressionOptionsIdentifiersForMediaType: and mediaType methods. Compression options are created from presets keyed by a named identifier. These preset identifiers are listed in the QTKit Framework Reference in the chapter describing this class.

Using QTSampleBuffer objects, you can get information about sample buffer data that you may need to output or process the media samples in the buffer.

User Interface Classes

There are two classes in this particular group, one devoted to previewing video, another providing support for Core Animation layers. The QTCaptureView and QTCaptureLayer objects are shown in Table 2-4.

You can use the methods available in the QTCaptureView class, which is a subclass of NSView, to preview video that is being processed by an instance of QTCaptureSession. The class creates and maintains its own QTCaptureVideoPreviewOutput to gather the preview video you need from the capture session.

Support for Core Animation is provided by the QTCaptureLayer class, which is a subclass of CALayer.

Device Access Class

There is only one class in this particular group, which is devoted to accessing a device, as shown in Table 2-5.

If you’re working with QTCaptureDevice objects, your application can read any number of extended attributes available to this class, using the deviceAttributes and attributeForKey: methods. Beyond that, you can use key-value coding to get and set attributes. If you wish to observe changes for a given attribute, you can add a key-value observer where the key path is the attribute key. Note that you cannot create instances of QTCaptureDevice directly.

Creating a Simple Capture and Recording Application

If you want to build a simple capture and recording application, these are coding steps and techniques you need to follow. Basically, you work with three capture classes to accomplish this task:

• QTCaptureSession. The primary interface for capturing media streams. This object acts as a kind of traffic cop, negotiating and managing the connections between inputs and outputs.

• QTCaptureMovieFileOutput. This is an output destination for QTCaptureSession that writes captured media to QuickTime movie files.

• QTCaptureDeviceInput. This object provides an input source for media devices, such as cameras and microphones.

Using these three objects, along with their associated methods, your application can perform simple capture and recording of media and then output the recorded content of the media to movie files for playback as QuickTime movies.

You follow these steps, preferably in the order defined below, to construct the application, using Xcode and Interface Builder tools:

1. Create an Cocoa application project in Xcode named, for example, MyRecorder.

   @interface MyRecorder : NSObject

2. Declare instance variables that point to the capture objects.

   QTCaptureSession *mCaptureSession;

   QTCaptureMovieFileOutput *mCaptureMovieFileOutput;

   QTCaptureDeviceInput *mCaptureVideoDeviceInput;

   QTCaptureDeviceInput *mCaptureAudioDeviceInput;

3. Define an Interface Builder outlet, mCaptureView, that points to a QTCaptureView object.

   IBOutlet QTCaptureView *mCaptureView;

4. In your implementation file, do the following:

   • Create the capture session.

     mCaptureSession = [[QTCaptureSession alloc] init];

   • Find a video device and connect your inputs and outputs to the session.

     QTCaptureDevice *videoDevice = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeVideo];

     success = [videoDevice open:&error];

     BOOL success = NO;

     NSError *error;

5. Add the video device to the session as a device input.

   mCaptureVideoDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:videoDevice];

   success = [mCaptureSession addInput:mCaptureVideoDeviceInput error:&error];

6. Create the movie file output and add it to the session.

   mCaptureMovieFileOutput = [[QTCaptureMovieFileOutput alloc] init];

   success = [mCaptureSession addOutput:mCaptureMovieFileOutput error:&error];

   if (!success) {

   }

   [mCaptureMovieFileOutput setDelegate:self];

7. Associate your capture view in the UI with the session and start it running.

   [mCaptureView setCaptureSession:mCaptureSession];

   [mCaptureSession startRunning];

8. Specify the compression options for the size of your video and the quality of your audio output.

   NSEnumerator *connectionEnumerator = [[mCaptureMovieFileOutput connections] objectEnumerator];

   QTCaptureConnection *connection;

   while ((connection = [connectionEnumerator nextObject])) {

   NSString *mediaType = [connection mediaType];

   QTCompressionOptions *compressionOptions = nil;

   if ([mediaType isEqualToString:QTMediaTypeVideo]) {

   compressionOptions = [QTCompressionOptions compressionOptionsWithIdentifier:@"QTCompressionOptions240SizeH264Video"];

   } else if ([mediaType isEqualToString:QTMediaTypeSound]) {

   compressionOptions = [QTCompressionOptions compressionOptionsWithIdentifier:@"QTCompressionOptionsHighQualityAACAudio"];

   }

   [mCaptureMovieFileOutput setCompressionOptions:compressionOptions forConnection:connection];

   }

9. Implement start and stop actions, and specify the output destination for your recorded media, in this case a QuickTime movie (.mov) in your /Users/Shared folder.

   - (IBAction)startRecording:(id)sender

   {

   [mCaptureMovieFileOutput recordToOutputFileURL:[NSURL fileURLWithPath:@"/Users/Shared/My Recorded Movie.mov"]];

   }

   - (IBAction)stopRecording:(id)sender

   {

   [mCaptureMovieFileOutput recordToOutputFileURL:nil];

   }

10. Output your captured movie at a specified path and open your QuickTime movie at the path you’ve specified at /Users/Shared/My Recorded Movie.mov.

    - (void)captureOutput:(QTCaptureFileOutput *)captureOutput didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL forConnections:(NSArray *)connections dueToError:(NSError *)error

    {

    [[NSWorkspace sharedWorkspace] openURL:outputFileURL];

    }

After building and compiling this code, your captured output appears as a movie playing in the QuickTime X Player application, as shown in Figure 2-5.

@"QTCompressionOptionsLosslessAppleIntermediateVideo"; //Not available in 64-bit.

@"QTCompressionOptionsLosslessAnimationVideo";

@"QTCompressionOptionsJPEGVideo";

@"QTCompressionOptions120SizeH264Video";

@"QTCompressionOptions240SizeH264Video";

@"QTCompressionOptionsSD480SizeH264Video";

@"QTCompressionOptions120SizeMPEG4Video"; //Not available in 64-bit.

@"QTCompressionOptions240SizeMPEG4Video"; //Not available in 64-bit.

@"QTCompressionOptionsSD480SizeMPEG4Video"; //Not available in 64-bit.

@"QTCompressionOptionsLosslessALACAudio";

@"QTCompressionOptionsHighQualityAACAudio";

@"QTCompressionOptionsVoiceQualityAACAudio";

- (NSSize)maximumVideoSize

- (void)setMaximumVideoSize:(NSSize)maximumVideoSize

Creating a Single-Frame Grabbing Application

If your application needs to capture single frames of media content—for example, from an internal or external iSight camera—you can do so easily and with a minimum lines of Cocoa code, using the capture classes in the QTKit API.

The technique of single-frame capture is common in the motion picture and television industries and is known as stop motion animation. You grab single frames from a video stream and output those frames, with great accuracy and reliability (avoiding tearing, for example), into a QuickTime movie. When the frames are concatenated into a movie for playback, you create the illusion of motion, and thus, the animation of still images.

To build this type of application, you work essentially with three capture classes:

• QTCaptureSession. This object is, again, your primary interface for capturing media streams.

• QTCaptureDecompressedVideoOutput. This is your output destination for a QTCaptureSession object that you can use to process decompressed frames from the video that is being captured. Using the methods provided in this class, you can produce decompressed video frames (CVImageBuffer), as shown in Figure 2-6, that are suitable for high-quality video processing.

• QTCaptureDeviceInput. This object is your input source for media devices, such as cameras and microphones.

You follow these steps, as outlined below, to construct this application:

1. Create a Cocoa-document based project in Xcode.

2. Declare three instance variables that point to a QTMovie object, one of which point to Interface Builder outlets.

   IBOutlet QTCaptureView *mCaptureView;

   IBOutlet QTMovieView *mMovieView;

   QTMovie *mMovie;

3. Declare instance variables pointing to three objects.

   QTCaptureSession *mCaptureSession;

   QTCaptureDeviceInput *mCaptureDeviceInput;

   QTCaptureDecompressedVideoOutput *mCaptureDecompressedVideoOutput;

4. Declare an IBAction method.

   - (IBAction)addFrame:(id)sender;

5. Set up a capture session that outputs the raw frames you want to grab.

   [mMovieView setMovie:mMovie];

   if (!mCaptureSession) {

   BOOL success;

   mCaptureSession = [[QTCaptureSession alloc] init];

6. Add a device input for that device to the capture session.

   QTCaptureDevice *device = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeVideo];

   success = [device open:&error];

   if (!success) {

   [[NSAlert alertWithError:error] runModal];

   return;

   }

   mCaptureDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:device];

   success = [mCaptureSession addInput:mCaptureDeviceInput error:&error];

   if (!success) {

   [[NSAlert alertWithError:error] runModal];

   return;

   }

7. Add a decompressed video output that returns raw frames to the session.

   mCaptureDecompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];

   [mCaptureDecompressedVideoOutput setDelegate:self];

   success = [mCaptureSession addOutput:mCaptureDecompressedVideoOutput error:&error];

   if (!success) {

   [[NSAlert alertWithError:error] runModal];

   return;

   }

8. Preview the video from the session in the document window.

   [mCaptureView setCaptureSession:mCaptureSession];

9. Start the session.

   [mCaptureSession startRunning];

   }

   }

10. Implement a delegate method that QTCaptureDecompressedVideoOutput calls whenever it receives a frame. Note that this method is called for each frame that comes in.

    - (void)captureOutput:(QTCaptureOutput *)captureOutput didOutputVideoFrame:(CVImageBufferRef)videoFrame withSampleBuffer:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection

11. Store the latest frame. You must perform this in a @synchronized block because this delegate method is not called on the main thread.

    {

    CVImageBufferRef imageBufferToRelease;

    CVBufferRetain(videoFrame);

    @synchronized (self) {

    imageBufferToRelease = mCurrentImageBuffer;

    mCurrentImageBuffer = videoFrame;

    }

    CVBufferRelease(imageBufferToRelease);

    }

12. Get the most recent frame. You must perform this in a @synchronized block because the delegate method that sets the most recent frame is not called on the main thread.

    - (IBAction)addFrame:(id)sender

    {

    CVImageBufferRef imageBuffer;

    @synchronized (self) {

    imageBuffer = CVBufferRetain(mCurrentImageBuffer);

    }

13. Create an NSImage and add it to the movie.

    if (imageBuffer) {

    NSCIImageRep *imageRep = [NSCIImageRep imageRepWithCIImage:[CIImage imageWithCVImageBuffer:imageBuffer]];

    NSImage *image = [[[NSImage alloc] initWithSize:[imageRep size]] autorelease];

    [image addRepresentation:imageRep];

    CVBufferRelease(imageBuffer);

    [mMovie addImage:image forDuration:QTMakeTime(1, 10) withAttributes:[NSDictionary dictionaryWithObjectsAndKeys:

    @"jpeg", QTAddImageCodecType,

    nil]];

    [mMovie setCurrentTime:[mMovie duration]];

    [mMovieView setNeedsDisplay:YES];

    [self updateChangeCount:NSChangeDone];

    }

    }

Pausing and Resuming Recording

If your application needs to control recording of captured media to a single file, you can take advantage of the enhanced recording functionality provided in OS X v10.6. You use the instance methods available in the QTCaptureFileOutput class that are designed to accomplish this task:

- (BOOL)isRecordingPaused

- (void)pauseRecording

- (void)resumeRecording

You use the isRecordingPaused method to return whether recording to the file returned by outputFileURL has been previously paused using the pauseRecording method. When you pause a recording, your captured samples are not written to the output file, but new samples can be written to the same file in the future by calling resumeRecording. The value of this method is key value observable using the key @"recordingPaused".

The pauseRecording method, as you would expect, causes the receiver to stop writing captured samples to the current output file returned by outputFileURL. However, it leaves the file open so that samples can be written to it in the future, when resumeRecording is called. This allows you to record multiple media segments that are not contiguous in time to a single file. When you stop recording or change files using recordToOutputFileURL:bufferDestination: or recording automatically stops due to an error condition while recording is paused, the output file will be finished and closed normally without requiring a matching call to resumeRecording. When there is no current output file, or when recording is already paused, this method does nothing. You can call this method within the captureOutput:didOutputSampleBuffer:fromConnection: delegate method to pause recording after an exact media sample.

Dealing with Dropped Late Video Frames

By default, the QTCaptureDecompressedVideoOutput object, a subclass of QTCaptureOutput, handles captured video frames via a delegate method, and sends all captured frames to the delegate, even if they are captured at a higher rate than the delegate is servicing them. If you don’t implement your own threading and frame-dropping strategy so that you don’t block the delegate thread on which you receive new video frames, your application will take a substantial performance hit, as video frames are queued up in memory more quickly than they can be serviced.

But if you don’t need to successfully capture every frame, you can use two methods available in QTCaptureDecompressedVideoOutput that allow you to automatically drop backed up, or late video frames. These methods are

- (BOOL)automaticallyDropsLateVideoFrames

- (void)setAutomaticallyDropsLateVideoFrames:(BOOL)automaticallyDropsLateVideoFrames

If you set the setAutomaticallyDropsLateVideoFrames: method to YES, the QTCaptureDecompressedVideoOutput object will discard extra frames that are queued up while you are blocking the delegate callback thread. In conjunction with this property, you can use the following delegate method:

- (void)captureOutput:(QTCaptureOutput *)captureOutput didDropVideoFrameWithSampleBuffer:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection;

When you set automaticallyDropsLateVideoFrames to YES, this method is called whenever a late video frame is dropped. The method is called once for each dropped frame and may be called before the call to the outputVideoFrame:withSampleBuffer:fromConnection: or the captureOutput:didOutputVideoFrame:withSampleBuffer:fromConnection: delegate method during which those frames were dropped returns. The QTSampleBuffer object passed to this delegate method will contain metadata about the dropped video frame, such as its duration and presentation time stamp, but contains no actual video data. Delegates should not assume that this method will be called on the main thread. Because this method may be called on the same thread that is responsible for outputting video frames, it must be efficient to prevent further capture performance problems, such as additional dropped video frames.

Controlling the Frame Rate of Captured Video

Many QTKit capture applications need to control the frame rate of captured video largely for performance reasons. For example, you may need to record video at a high resolution but don’t need to record at a high frame rate, so you can save a substantial amount of memory, disk bandwidth, and device bus bandwidth by avoiding capturing the extra frames. Two methods, in particular, both in QTCaptureFileOutput and QTCaptureDecompressedVideoOutput, are useful handling the responsibilities for outputting captured video. They are:

- (NSTimeInterval)minimumVideoFrameInterval

- (void)setMinimumVideoFrameInterval:(NSTimeInterval)minimumVideoFrameInterval

You use these methods to specify the minimum amount of time that should separate consecutive frames output by the receiver, which is the inverse of the maximum frame rate. In each case, a value of 0 indicates an unlimited maximum frame rate. The default value is 0. Whenever possible, devices that support different hardware frame rates will be automatically reconfigured to use the least bandwidth while still fulfilling the requirements of all outputs in the capture session.

Capturing a Single Stream from a Muxed Device

Muxed devices such as DV and HDV cameras, where audio and video streams are mixed, are represented in the QTKit capture API by the media type QTMediaTypeMuxed. To capture a single stream from the device (for example, video only), you disable the audio connections on the device input for the capture device by using the QTCaptureConnection setEnabled: method and passing in NO.

In the QTKit capture API, the QTCaptureDevice object, shown in Listing 2-2, represents an available capture device. This means that each instance of QTCaptureDevice corresponds to a capture device that is connected or has been previously connected to the user’s computer during the lifetime of the application. You cannot create instances of QTCaptureDevice directly. A single unique instance is created automatically whenever a device is connected to the computer and can be accessed using the deviceWithUniqueID: class method. An array of all currently connected devices can also be obtained using the inputDevices class method.

Table 2-6 details the media types supported by QTCaptureDevice and examples of devices that support them:

Listing 2-2 shows how to capture a single stream from a muxed device.

Listing 2-2  Capturing a single stream from a muxed device

QTCaptureDevice *theDefaultMuxedDevice;

QTCaptureDeviceInput *theDeviceInput;

BOOL success;

NSError *error;

...

// get the default muxed device

theDefaultMuxedDevice = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeMuxed];

// open the device

success = [theDefaultMuxedDevice open:&error];

if (YES == success) {

    // create and associate device input

    theDeviceInput = [QTCaptureDeviceInput deviceInputWithDevice:theDefaultMuxedDevice];

    // get the list of owned connections

    NSArray *ownedConnections = [theDeviceInput connections];

    // disable all the audio connections

    for (QTCaptureConnection *connection in ownedConnections) {

        if ( [[connection mediaType] isEqualToString:QTMediaTypeSound] ) {

            [connection setEnabled:NO];

        }

    }

} else {

    // do something with the error code

}

Configuring Cameras for Capture

If you want to configure your camera directly for capture, QTKit will allow you to specify the output and configure the camera if it can.

QTKit will automatically adjust the camera resolution to best accommodate the requirements of all the outputs connected to a particular session. For example, if you set the compression options of a movie file output or the pixel buffer attributes of a decompressed video output, you can cause the camera resolution to be adjusted for optimal performance. In general, if you are interested in configuring the camera because you want your output video to be at a specific resolution, it makes more sense to configure your outputs directly and not worry about the camera settings, which is the functionality currently provided by QTKit.

In the use case where you need to access the captured frames’ pixel values for pattern recognition to resize the CVImageBuffer, QTKit provides an appropriate API. You can ask for CVImageBuffers of a specific size and pixel format using QTCaptureDecompressedVideoOutput setPixelBufferAttributes:. This is illustrated in the following usage example, shown in Listing 2-3.

Listing 2-3  Capturing decompressed video output frames

    [decompressedVideoOutput setPixelBufferAttributes:[NSDictionary

                                                        dictionaryWithObjectsAndKeys:

                                                       [NSNumber

                                                        numberWithDouble:320.0], (id)kCVPixelBufferWidthKey,

                                                       [NSNumber

                                                        numberWithDouble:240.0], (id)kCVPixelBufferHeightKey,

                                                       [NSNumber

                                                        numberWitUnsignedInt:kCVPixelFormatType_32ARGB],

                                                        (id)kCVPixelBufferPixelFormatTypeKey,

                                                       nil]];

When appropriate, specifying these pixel buffer attributes will cause the camera to be automatically reconfigured to best match the output format. Conversions still may be necessary. For example, if the camera does not have an RGB mode, then a pixel format conversion will still be required. However, QTKit will perform all conversions for you. In short, using this API allows QTKit to choose the most optimal video pipeline for the requested output.