Time and Media Representations

Representation of Assets

AVAsset is the core class in the AV Foundation framework. It provides a format-independent abstraction of time-based audiovisual data, such as a movie file or a video stream. In many cases, you work with one of its subclasses: you use the composition subclasses when you create new assets (see “Editing”), and you use AVURLAsset to create a new asset instance from media at a given URL (including assets from the MPMedia framework or the Asset Library framework—see “Using Assets”).

An asset contains a collection of tracks that are intended to be presented or processed together, each of a uniform media type, including (but not limited to) audio, video, text, closed captions, and subtitles. The asset object provides information about whole resource, such as its duration or title, as well as hints for presentation, such as its natural size. Assets may also have metadata, represented by instances of AVMetadataItem.

A track is represented by an instance of AVAssetTrack. In a typical simple case, one track represents the audio component and another represents the video component; in a complex composition, there may be multiple overlapping tracks of audio and video.

A track has a number of properties, such as its type (video or audio), visual and/or audible characteristics (as appropriate), metadata, and timeline (expressed in terms of its parent asset). A track also has an array of format descriptions. The array contains CMFormatDescriptions (see CMFormatDescriptionRef), each of which describes the format of media samples referenced by the track. A track that contains uniform media (for example, all encoded using to the same settings) will provide an array with a count of 1.

A track may itself be divided into segments, represented by instances of AVAssetTrackSegment. A segment is a time mapping from the source to the asset track timeline.

Representations of Time

Time in AV Foundation is represented by primitive structures from the Core Media framework.

CMTime time1 = CMTimeMake(200, 2); // 200 half-seconds

CMTime time2 = CMTimeMake(400, 4); // 400 quarter-seconds

// time1 and time2 both represent 100 seconds, but using different timescales.

if (CMTimeCompare(time1, time2) == 0) {

    NSLog(@"time1 and time2 are the same");

}

Float64 float64Seconds = 200.0 / 3;

CMTime time3 = CMTimeMakeWithSeconds(float64Seconds , 3); // 66.66... third-seconds

time3 = CMTimeMultiply(time3, 3);

// time3 now represents 200 seconds; next subtract time1 (100 seconds).

time3 = CMTimeSubtract(time3, time1);

CMTimeShow(time3);

if (CMTIME_COMPARE_INLINE(time2, ==, time3)) {

    NSLog(@"time2 and time3 are the same");

}

CMTime myTime = <#Get a CMTime#>;

if (CMTIME_IS_INVALID(myTime)) {

    // Perhaps treat this as an error; display a suitable alert to the user.

}

CMTimeRangeContainsTime(range, CMTimeRangeGetEnd(range))

CMTimeRange myTimeRange = <#Get a CMTimeRange#>;

if (CMTIMERANGE_IS_EMPTY(myTimeRange)) {

    // The time range is zero.

}

Representations of Media

Video data and its associated metadata is represented in AV Foundation by opaque objects from the Core Media framework. Core Media represents video data using CMSampleBuffer (see CMSampleBufferRef). CMSampleBuffer is a Core Foundation-style opaque type; an instance contains the sample buffer for a frame of video data as a Core Video pixel buffer (see CVPixelBufferRef). You access the pixel buffer from a sample buffer using CMSampleBufferGetImageBuffer:

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(<#A CMSampleBuffer#>);

From the pixel buffer, you can access the actual video data. For an example, see “Converting a CMSampleBuffer to a UIImage.”

In addition to the video data, you can retrieve a number of other aspects of the video frame:

• Timing information You get accurate timestamps for both the original presentation time and the decode time using CMSampleBufferGetPresentationTimeStamp and CMSampleBufferGetDecodeTimeStamp respectively.

• Format information The format information is encapsulated in a CMFormatDescription object (see CMFormatDescriptionRef). From the format description, you can get for example the pixel type and video dimensions using CMVideoFormatDescriptionGetCodecType and CMVideoFormatDescriptionGetDimensions respectively.

• Metadata Metadata are stored in a dictionary as an attachment. You use CMGetAttachment to retrieve the dictionary:

  CMSampleBufferRef sampleBuffer = <#Get a sample buffer#>;

  CFDictionaryRef metadataDictionary =

  CMGetAttachment(sampleBuffer, CFSTR("MetadataDictionary", NULL);

  if (metadataDictionary) {

  // Do something with the metadata.

  }

Converting a CMSampleBuffer to a UIImage

The following function shows how you can convert a CMSampleBuffer to a UIImage object. You should consider your requirements carefully before using it. Performing the conversion is a comparatively expensive operation. It is appropriate to, for example, create a still image from a frame of video data taken every second or so. You should not use this as a means to manipulate every frame of video coming from a capture device in real time.

UIImage *imageFromSampleBuffer(CMSampleBufferRef sampleBuffer) {

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    // Lock the base address of the pixel buffer.

    CVPixelBufferLockBaseAddress(imageBuffer,0);

    // Get the number of bytes per row for the pixel buffer.

    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);

    // Get the pixel buffer width and height.

    size_t width = CVPixelBufferGetWidth(imageBuffer);

    size_t height = CVPixelBufferGetHeight(imageBuffer);

    // Create a device-dependent RGB color space.

    static CGColorSpaceRef colorSpace = NULL;

    if (colorSpace == NULL) {

        colorSpace = CGColorSpaceCreateDeviceRGB();

            if (colorSpace == NULL) {

            // Handle the error appropriately.

            return nil;

        }

    }

    // Get the base address of the pixel buffer.

    void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);

    // Get the data size for contiguous planes of the pixel buffer.

    size_t bufferSize = CVPixelBufferGetDataSize(imageBuffer);

    // Create a Quartz direct-access data provider that uses data we supply.

    CGDataProviderRef dataProvider =

        CGDataProviderCreateWithData(NULL, baseAddress, bufferSize, NULL);

    // Create a bitmap image from data supplied by the data provider.

    CGImageRef cgImage =

        CGImageCreate(width, height, 8, 32, bytesPerRow,

                        colorSpace, kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrder32Little,

                        dataProvider, NULL, true, kCGRenderingIntentDefault);

    CGDataProviderRelease(dataProvider);

    // Create and return an image object to represent the Quartz image.

    UIImage *image = [UIImage imageWithCGImage:cgImage];

    CGImageRelease(cgImage);

    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    return image;

}