Representation of Assets

AVAsset is the core class in the AV Foundation framework. It provides a format-independent abstraction of time-based audiovisual data, such as a movie file or a video stream. The primary relationships are shown in Figure 6-1. In many cases, you work with one of its subclasses: You use the composition subclasses when you create new assets (see Editing), and you use AVURLAsset to create a new asset instance from media at a given URL (including assets from the MPMedia framework or the Asset Library framework—see Using Assets).

An asset contains a collection of tracks that are intended to be presented or processed together, each of a uniform media type, including (but not limited to) audio, video, text, closed captions, and subtitles. The asset object provides information about whole resource, such as its duration or title, as well as hints for presentation, such as its natural size. Assets may also have metadata, represented by instances of AVMetadataItem.

A track is represented by an instance of AVAssetTrack, as shown in Figure 6-2. In a typical simple case, one track represents the audio component and another represents the video component; in a complex composition, there may be multiple overlapping tracks of audio and video.

A track has a number of properties, such as its type (video or audio), visual and/or audible characteristics (as appropriate), metadata, and timeline (expressed in terms of its parent asset). A track also has an array of format descriptions. The array contains CMFormatDescription objects (see CMFormatDescriptionRef), each of which describes the format of media samples referenced by the track. A track that contains uniform media (for example, all encoded using to the same settings) will provide an array with a count of 1.

A track may itself be divided into segments, represented by instances of AVAssetTrackSegment. A segment is a time mapping from the source to the asset track timeline.

Representations of Time

Time in AV Foundation is represented by primitive structures from the Core Media framework.

CMTime time1 = CMTimeMake(200, 2); // 200 half-seconds

CMTime time2 = CMTimeMake(400, 4); // 400 quarter-seconds

// time1 and time2 both represent 100 seconds, but using different timescales.

if (CMTimeCompare(time1, time2) == 0) {

    NSLog(@"time1 and time2 are the same");

}

Float64 float64Seconds = 200.0 / 3;

CMTime time3 = CMTimeMakeWithSeconds(float64Seconds , 3); // 66.66... third-seconds

time3 = CMTimeMultiply(time3, 3);

// time3 now represents 200 seconds; next subtract time1 (100 seconds).

time3 = CMTimeSubtract(time3, time1);

CMTimeShow(time3);

if (CMTIME_COMPARE_INLINE(time2, ==, time3)) {

    NSLog(@"time2 and time3 are the same");

}

CMTime myTime = <#Get a CMTime#>;

if (CMTIME_IS_INVALID(myTime)) {

    // Perhaps treat this as an error; display a suitable alert to the user.

}

CMTimeRangeContainsTime(range, CMTimeRangeGetEnd(range))

CMTimeRange myTimeRange = <#Get a CMTimeRange#>;

if (CMTIMERANGE_IS_EMPTY(myTimeRange)) {

    // The time range is zero.

}

Representations of Media

Video data and its associated metadata are represented in AV Foundation by opaque objects from the Core Media framework. Core Media represents video data using CMSampleBuffer (see CMSampleBufferRef). CMSampleBuffer is a Core Foundation-style opaque type; an instance contains the sample buffer for a frame of video data as a Core Video pixel buffer (see CVPixelBufferRef). You access the pixel buffer from a sample buffer using CMSampleBufferGetImageBuffer:

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(<#A CMSampleBuffer#>);

From the pixel buffer, you can access the actual video data. For an example, see Converting CMSampleBuffer to a UIImage Object.

In addition to the video data, you can retrieve a number of other aspects of the video frame:

• Timing information. You get accurate timestamps for both the original presentation time and the decode time using CMSampleBufferGetPresentationTimeStamp and CMSampleBufferGetDecodeTimeStamp respectively.

• Format information. The format information is encapsulated in a CMFormatDescription object (see CMFormatDescriptionRef). From the format description, you can get for example the pixel type and video dimensions using CMVideoFormatDescriptionGetCodecType and CMVideoFormatDescriptionGetDimensions respectively.

• Metadata. Metadata are stored in a dictionary as an attachment. You use CMGetAttachment to retrieve the dictionary:

  CMSampleBufferRef sampleBuffer = <#Get a sample buffer#>;

  CFDictionaryRef metadataDictionary =

  CMGetAttachment(sampleBuffer, CFSTR("MetadataDictionary", NULL);

  if (metadataDictionary) {

  // Do something with the metadata.

  }

Converting CMSampleBuffer to a UIImage Object

The following code shows how you can convert a CMSampleBuffer to a UIImage object. You should consider your requirements carefully before using it. Performing the conversion is a comparatively expensive operation. It is appropriate to, for example, create a still image from a frame of video data taken every second or so. You should not use this as a means to manipulate every frame of video coming from a capture device in real time.

// Create a UIImage from sample buffer data

- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer

{

    // Get a CMSampleBuffer's Core Video image buffer for the media data

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    // Lock the base address of the pixel buffer

    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    // Get the number of bytes per row for the pixel buffer

    void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);

    // Get the number of bytes per row for the pixel buffer

    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);

    // Get the pixel buffer width and height

    size_t width = CVPixelBufferGetWidth(imageBuffer);

    size_t height = CVPixelBufferGetHeight(imageBuffer);

    // Create a device-dependent RGB color space

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

    // Create a bitmap graphics context with the sample buffer data

    CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,

      bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);

    // Create a Quartz image from the pixel data in the bitmap graphics context

    CGImageRef quartzImage = CGBitmapContextCreateImage(context);

    // Unlock the pixel buffer

    CVPixelBufferUnlockBaseAddress(imageBuffer,0);

    // Free up the context and color space

    CGContextRelease(context);

    CGColorSpaceRelease(colorSpace);

    // Create an image object from the Quartz image

    UIImage *image = [UIImage imageWithCGImage:quartzImage];

    // Release the Quartz image

    CGImageRelease(quartzImage);

    return (image);

}