Video Enhancements
QuickTime 7 for Windows introduces a number of important video enhancements, discussed in this section. These include
Support for compressed video using frame reordering. Support is added for compression, playback, streaming, and low-level access to stored samples.
Support for H.264 video compression, including QuickTime components for export, playback, and live streaming.
In this section:
Frame Reordering Video
H.264 Codec
Frame Reordering Video
QuickTime 7 for Windows adds support for frame reordering video compression. This is a major advance that involves new sample tables for video to allow video frames to have independent decode and display times.
The result of using frame reordering for video compression is improved display and editing in H.264 and other advanced video codec formats. Enhancements include a new API for working with media sample times, adding and finding samples, and a new Image Compression Manager (ICM) API.
Understanding Frame Reordering Video Compression
QuickTime supports many types of video compression, including spatial compression algorithms, such as photo-JPEG, and temporal compression algorithms, in which some video frames are described completely, while other frames are described in terms of their differences from other video frames.
Up until the introduction of H.264 in QuickTime 7 for Windows, video frames could be of three kinds:
I-frames (independently decodable)
P-frames (predicted from a previous I- or P-frame)
B-frames (predicted from one past and one future I- or P-frame)
Note: B-frame, I-frame, and P-frame are all video compression methods used by the MPEG standard. B-frame is an abbreviation for bi-directional frame, or bi-directional predictive frame. B-frames rely on the frames preceding and following them and only contain data that has changed from the preceding frame or is different from data in the next frame.
P-frame is an abbreviation for predictive frame, or predicted frame. P-frames follow I-frames and contain only the data that has changed from the preceding I-frame. P-frames rely on I-frames to fill in most of its data.
I-frame, also known as keyframes, is an abbreviation for intraframe. An I-frame stores all the data required to display the frame. In common usage, I-frames are interspersed with P-frames and B-frames in a compressed video.
Because B-frames predict from a future frame, that frame has to be decoded before the B-frame, yet displayed after it; this is why frame reordering is needed.The decoded order is no longer the same as the displayed order.The QuickTime support for frame reordering is quite general.In the H.264 codec, the concepts of the direction of prediction, and the numbers of referenced frames, and the kind of frame that is referenced, are all decoupled. In H.264, an encoder may choose to make a stream in which P-frames refer to a future frame, or a B-frame which refers to two past or future frames, for example.
Important: Prior to this release, QuickTime supported self-contained video frames (keyframes, also called sync-frames or I-frames) and frames that depended on previous frames (P-frames). Many modern compressors also make use of frame reordering, in which frames can depend on future frames.Those future frames have to be decoded before the frame in question, but displayed after it––hence the reordering. Traditional B-frames are one example: they depend on a past and a future I- or P-frame.That future I- or P-frame has to be given to the decoder before the B-frame, but is displayed after the B-frame itself. This means that the frames are stored or streamed in decode order, rather than in display order.
For decompressors that don’t use frame reorderings, the decode order and the display order are the same, and QuickTime sample tables are traditionally organized to reflect this. Samples are stored in decode order, which is presumed to be the display order, and the sample tables specify the duration of each sample’s display; the display time is the time when the track begins plus the duration of all previous samples.
The addition of frame reordering support means that QuickTime now has an optional sample table for video that specifies the offset between the decode time and the display time. This allows frames to be stored in decode order but displayed in a different order. The decode time is still the beginning of the track plus the decode duration of all previous samples, but it is now necessary to examine the offset table to determine which samples precede others and calculate the correct display time.
For high-level programmers, this all happens transparently. Developers who work directly with sample numbers and sample times, however, must be aware of this new feature. A new, expanded API is available to support this.
Finding and Adding Samples
Developers who need to work with specific samples based on the samples’ display times, or who are adding samples to a media directly, need to use a different API when working with media that uses frame reorderings.
For example, programmers who use the function MediaTimeToSampleNum must
instead use the two functions MediaDecodeTimeToSampleNum and MediaDisplayTimeToSampleNum when
working with frame reordering compressed video, as each sample now
has a decode time and a display time instead of a single media time
(combined decode/display time).
Similarly, when adding samples to a media that permits display
offsets, it is necessary to use the new AddMediaSample2 instead
of AddMediaSample, as the new function
permits the user to pass a display offset and specify properties
that are unique to media with display offsets, such as whether subsequent
samples are allowed to have earlier display times than the current
sample.
Calling one of the old functions that use a single media time
value on new-format media that contains display offsets will return
the error code kQTErrMediaHasDisplayOffsets.
The new API elements all use 64-bit time values, whereas the
older API elements use 32-bit values. Calling one of the old functions
with a 64-bit time value returns the error code kQTErrTimeValueTooBig.
When creating a media for frame reordering compressed video
track, pass in the new flag kCharacteristicSupportsDisplayOffsets.
For details, see:
There is additional support for programmers who work directly with arrays of media sample references. Although these new functions work with frame reordering video or other media with independent decode and display times, they can also be used with ordinary media types. See “QuickTime Sample Table API.”
Compressing Video Using Frame Reordering
When compressing video that uses frame reordering, there is no longer a one-to-one correspondence between submitting a frame for compression and getting back a compressed sample. The Image Compression Manager (ICM) and the compressor component may buffer multiple images before determining that a series of frames should be B-frames and a subsequent image should be decompressed out of order so that the B-frames can refer to it. The new ICM functions do not require a strict correlation between input frames and output frames. Frames may be rearranged by compression and decompression modules.
The new functions allow groups of multiple pixel buffers to
be in use at various processing points in order to avoid unnecessary
copying of data, using CVPixelBuffers and CVPixelBufferPools.
These new types are Core Foundation based. They follow Core Foundation’s
protocols for reference counting (create/copy/retain/release). Each
type has its own retain and release functions which are type-safe
and NULL-safe, but otherwise equivalent to CFRetain and CFRelease.
Note that the CVPixelBuffer functions generally provide their output
data through callbacks, rather than as return values or function parameters.
In general, the new functions return OSStatus,
with the exception of some simple Get functions that return single
values.
Clients create compression sessions using ICMCompressionSessionCreate. They
then feed pixel buffers in display order to ICMCompressionSessionEncodeFrame.
Encoded frames may not be output immediately, and may not be returned
in the same order as they are input—encoded frames will be returned
in decode order, which will sometimes differ from display order.
One of the parameters to ICMCompressionSessionCreate specifies
a callback routine that QuickTime will call when each encoded frame
is ready. Frames should be stored in the order they are output (decode order).
To force frames up to a certain display time to be encoded
and output, call ICMCompressionSessionCompleteFrames.
To obtain a pixel buffer pool that satisfies the requirements
of both your pixel buffer producer and the compressor, pass the
pixel buffer producer’s pixel buffer options into ICMCompressionSessionCreate,
and then call ICMCompressionSessionGetPixelBufferPool. The compression session constructs an appropriate pixel
buffer pool.
Alternatively, you can create your own pixel buffer pool by obtaining the compressor’s pixel buffer attributes, choosing a format compatible with your pixel buffer producer, and setting that compressor’s input format using the component properties API. The process of obtaining the pixel buffer attributes is illustrated in the following code snippet.
CFDictionaryRef attributesDictionary = NULL; |
err = ICMCompressionSessionGetProperty( |
session, |
kQTPropertyClass_ICMCompressionSession, |
kICMCompressionSessionPropertyID_CompressorPixelBufferAttributes, |
sizeof(CFDictionaryRef), |
&attributesDictionary, |
NULL); |
if (attributesDictionary) { |
// ...use... |
CFRelease(attributesDictionary); |
} |
You can also pass arbitrary pixel buffers to ICMCompressionSessionEncodeFrame;
if they’re incompatible with the compressor’s requirements,
then the compression session will make compatible copies and pass
those to the compressor. This requires less setup but can result in
significantly slower operation.
When the compressor no longer needs a source pixel buffer,
it will release it. You may also pass ICMCompressionSessionEncodeFrame a
callback to be called when the source pixel buffer is released.
Clients may call ICMCompressionSessionGetImageDescription to get the image description for the encoded frames.
Where possible, the ICM will allow this to be called before the
first frame is encoded.
For additional details, see:
H.264 Codec
The H.264 codec is the latest standards-based video codec. Published jointly by the ITU as H.264––Advanced Video Coding, and by ISO as MPEG-4 Part 10––Advanced Video Coding, the H.264 codec promises better image quality at lower bit rates than the current MPEG-4 video codec, and also better live streaming characteristics than the current H.263 codec.
This represents a significant increase in quality and performance, while operating in a standards-based framework.
QuickTime 7 for Windows includes a QuickTime decompressor component and an exporter component for creating and playing H.264-encoded video in QuickTime.
The H.264 codec makes use of QuickTime 7’s new support for frame reordering video compression.
Last updated: 2005-11-09
  |
||
|
|
|
|
| ||
Get information on Apple products.
Visit the Apple Store online or at retail locations. 1-800-MY-APPLE Copyright © 2007 Apple Inc. All rights reserved. | Terms of use | Privacy Notice |