Video Decode Acceleration Framework Reference
This reference describes the Video Decode Acceleration framework available on Mac OS X v10.6.3 and later with Mac models equipped with the NVIDIA GeForce 9400M, GeForce 320M, GeForce GT 330M, ATI HD Radeon GFX, Intel HD Graphics and others.
Overview
Framework VideoDecodeAcceleration Header VDADecoder.hThe Video Decode Acceleration framework is a C programming interface providing low-level access to the H.264 decoding capabilities of compatible GPUs such as the NVIDIA GeForce 9400M, GeForce 320M, GeForce GT 330M, ATI HD Radeon GFX, Intel HD Graphics and others. It is intended for use by advanced developers who specifically need hardware accelerated decode of video frames.
The framework allows you to:
Create a
VDADecoder
object which functions as the interface to hardware decode resources.Provide an output callback to receive a decompressed image buffer once decoding is complete.
Pass in H.264 compressed video frames, one at a time to a
VDADecoder
object for decode.Cancel decompression of all non-decoded frames currently in flight.
Destroy a
VDADecoder
object releasing hardware resources.
The Video Decode Acceleration framework is available on Mac OS X 10.6.3 and later.
Functions
VDADecoderCreate
Creates a new VDADecoder
object.
OSStatus VDADecoderCreate(CFDictionaryRef decoderConfiguration, |
CFDictionaryRef destinationImageBufferAttributes, |
VDADecoderOutputCallback *outputCallback, |
void *decoderOutputCallbackRefcon, |
VDADecoder *decoderOut) |
Parameters
decoderConfiguration
- ACFDictionaryRef
containingkVDADecoderConfiguratioXXX
keys describing the source data and configuration for the decoder. See Decoder Configuration Keys.destinationImageBufferAttributes
- ACFDictionaryRef
describing the clients requirements for output image buffers. This parameter may beNULL
if the client has no specific preference for the format of the output image buffers. IfNULL
is used, the client should make no assumptions regarding the returned image buffers. See Destination Image Buffer Attribute Keys.outputCallback
- AVDADecoderOutputCallback
function called by the decoder when returning decompressed image buffers to the client.decoderOutputCallbackRefcon
- A pointer to user data passed to the output callback for all frames from this decoder object.decoderOut
- On output, the newly created decoder object.
Discussion
Creates an interface for utilizing hardware resources to decode video by returning a VDADecoder
object. If hardware doesn't exist to decode the provided format or if there are insufficient hardware resources to perform hardware decoding, the appropriate result code will be returned.
Decoded frames are emitted via the VDADecoderOutputCallback
function passed in by the caller.
CVPixelBufferGetPixelFormatType
may be used to verify the pixel format of the returned image buffers to ensure the image buffer contains the format you are prepared to handle. It is good practice to verify the image buffer format before doing any format-specific processing.
Since there is no automatic software fallback when using the Video Decode Acceleration framework or an implicit guarantee that you will be able to access hardware accelerated decode (even on supported configurations), the client will need to respond accordingly if the attempt to create a decoder object fails and provide its own fallback strategy.
Return Value
A result code. See Result Codes.
VDADecoderDestroy
Releases the decoder object.
OSStatus VDADecoderDestroy(VDADecoder decoder) |
Parameters
decoder
- The hardware decoder object being destroyed.
Discussion
This call frees the decoder object and releases all resources currently in use by the decoder. All queued frames will be flushed without invoking the output callback. The decoder instance should no longer be referenced after calling VDADecoderDestroy
.
Return Value
A result code. See Result Codes.
VDADecoderDecode
Send the included compressed data to the hardware decoder object for decoding.
OSStatus VDADecoderDecode(VDADecoder decoder, |
uint32_t decodeFlags, |
CFTypeRef compressedBuffer, |
CFDictionaryRef frameInfo) |
Parameters
decoder
- The hardware decoder object performing the decode operation.decodeFlags
- Flags containing any special requests for this decode operation. See Decode Flags.compressedBuffer
- ACFDataRef
containing a single H.264 compressed frame to be decoded.frameInfo
- ACFDictionaryRef
containing information to be returned in the output callback for this frame. This dictionary can contain client provided information associated with the frame being decoded, for example presentation time. TheCFDictionaryRef
will be retained by the framework.
Discussion
This call will send a single H.264 compressed frame packaged as a CFDataRef
to the hardware decoder object for decoding. The decoded frame is output via the output callback.
Return Value
A result code. See Result Codes.
VDADecoderFlush
Flush all frames currently being decoded by the hardware decoder.
OSStatus VDADecoderFlush(VDADecoder decoder, uint32_t flushFlags) |
Parameters
decoder
- The hardware decoder object performing the decode operation.flushFlags
- Flags controlling flush behavior, for example, whether or not currently in flight decode operations will emit frames. See Flush Flags.
Discussion
Flushing will cancel all currently queued frames which have not been completed. As the hardware decoder is generally running asynchronously, there is the possibility of a frame completing while the call to flush is in progress. No completed frames should be returned after control returns from this call.
The output callback is still called for all flushed frames, but no image buffers will be returned.
If the kVDADecoderFlush_emitFrames
flag is specified, the flush operation will return image buffers in the output callback.
Return Value
A result code. Result Codes.
Output Callback
Prototype for a callback function invoked when a frame is decoded by calling VDADecoderDecode
.
typedef void (*VDADecoderOutputCallback)(void *decompressionOutputRefCon, |
CFDictionaryRef frameInfo, |
OSStatus status, |
uint32_t infoFlags, |
CVImageBufferRef imageBuffer) |
Parameters
decompressionOutputRefCon
- The user data pointer as passed intoVDADecoderCreate
fordecoderOutputCallbackRefcon
.frameInfo
- TheframeInfo
dictionary passed toVDADecoderDecode
for this frame. TheCFDictionaryRef
will be released after returning from the callback. The client must retain it if it is needed beyond the scope of the output callback.status
- An error code is returned if decompression was not successful. See Result Codes.infoFlags
- Contains information about a decode operation. For example, thekVDADecodeInfo_FrameDropped
flag may be set if the frame was dropped. See Decode Info Flags.imageBuffer
- A Core Video image buffer containing a decompressed video frame. The client should retain returned image buffers and release them when they are no longer needed. Image buffers are NOT owned by the decoder instance.
Discussion
When creating a VDADecoder
object, pass in a VDADecoderOutputCallback
function to be used to return decompressed frames. The output callback may be called in decode order rather than presentation order.
Data Types
VDADecoder
OpaqueVDADecoder* VDADecoder |
A reference to a video decoder object.
Constants
Decoder Configuration Keys
kVDADecoderConfiguration_Height
- ACFNumberRef
specifying source height.kVDADecoderConfiguration_Width
- ACFNumberRef
specifying source width.kVDADecoderConfiguration_SourceFormat
- ACFNumberRef
(kCFNumberSInt32Type
Four Character Code) specifying source format. For example,'avc1'
.kVDADecoderConfiguration_avcCData
- ACFDataRef
containingavcC
data from the H.264 bitstream. In a QuickTime movie file, this is the same data which is stored in the image description as theavcC
atom.
Destination Image Buffer Attribute Keys
Listed are the commonly used keys, for a full list of available Core Video Pixel Buffer keys see CoreVideo/CVPixelBuffer.h
kCVPixelBufferPixelFormatTypeKey
- A singleCFNumberRef
(OSType) or an array ofCFNumberRef
(OSTypes) specifying output pixel format. For example,kCVPixelFormatType_422YpCbCr8
.kCVPixelBufferWidthKey
-ACFNumberRef
specifying the width of the pixel buffer.kCVPixelBufferHeightKey
- ACFNumberRef
specifying the height of the pixel buffer.kCVPixelBufferIOSurfacePropertiesKey
- ACFDictionaryRef
of IOSurface properties. The presence of this key requests buffer allocation via IOSurface. If adestinationImageBufferAttributes
dictionary is specified inVDADecoderCreate
, that dictionary MUST contain this key. The value for the this key is most often just an empty dictionary. IfNULL
is specified inVDADecoderCreate
for thedestinationImageBufferAttributes
dictionary, this key is assumed to be set. The hardware decoder requires IOSurface backed image buffers for rendering. SeeIOSurface/IOSurfaceAPI.h
for the complete list of available IOSurface properties.
Decode Flags
Flags used with the VDADecoderDecode
.
kVDADecoderDecodeFlags_DontEmitFrame = 1 << 0 |
Specifies that the decoder should not return an image buffer in the output callback. Note that the output callback is still invoked.
Flush Flags
Flags used with the VDADecoderFlush
.
kVDADecoderFlush_EmitFrames = 1 << 0 |
During a flush operation this flag specifies that the decoder should decode and return image buffers for all currently queued frames.
Decode Info Flags
During decode, flags may be set in the infoFlags
field of the output callback conveying extra information about a decode operation.
kVDADecodeInfo_Asynchronous = 1UL << 0 |
kVDADecodeInfo_FrameDropped = 1UL << 1 |
kVDADecodeInfo_Asynchronous
- Indicates asynchronous decoding. As asynchronous decode is the normal operating mode of the decoder object, this flag will always be set during normal operation.kVDADecodeInfo_FrameDropped
- Indicates the frame was dropped.
Result Codes
Errors returned from the VDADecoder
APIs.
kVDADecoderNoErr = 0 |
kVDADecoderHardwareNotSupportedErr = -12470 |
kVDADecoderFormatNotSupportedErr = -12471 |
kVDADecoderConfigurationError = -12472 |
kVDADecoderDecoderFailedErr = -12473 |
kVDADecoderNoErr
- Life is good.kVDADecoderHardwareNotSupportedErr
- The hardware does not support accelerated video services required for hardware decode.kVDADecoderFormatNotSupportedErr
- The hardware may support accelerated decode, but does not support the requested output format.kVDADecoderConfigurationError
- Invalid or unsupported configuration parameters were specified inVDADecoderCreate
.kVDADecoderDecoderFailedErr
- An error was returned by the decoder layer. This may happen for example because of bitstream/data errors during a decode operation. This error may also be returned fromVDADecoderCreate
when hardware decoder resources are available on the system but currently in use by another process.
Sample Code
The following data types and helper functions are referenced in the code listings below and presented for illustrative purposes only. They are not part of the framework.
// tracks a frame in and output queue in display order |
typedef struct myDisplayFrame { |
int64_t frameDisplayTime; |
CVPixelBufferRef frame; |
struct myDisplayFrame *nextFrame; |
} myDisplayFrame, *myDisplayFramePtr; |
// some user data |
typedef struct MyUserData |
{ |
... |
myDisplayFramePtr displayQueue; // display-order queue - next display frame is always at the queue head |
int32_t queueDepth; // we will try to keep the queue depth around 10 frames |
pthread_mutex_t queueMutex; // mutex protecting queue manipulation |
... |
} MyUserData, *MyUserDataPtr; |
// example helper function that wraps a time into a dictionary |
static CFDictionaryRef MakeDictionaryWithDisplayTime(int64_t inFrameDisplayTime) |
{ |
CFStringRef key = CFSTR("MyFrameDisplayTimeKey"); |
CFNumberRef value = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt64Type, &inFrameDisplayTime); |
return CFDictionaryCreate(kCFAllocatorDefault, |
(const void **)&key, |
(const void **)&value, |
1, |
&kCFTypeDictionaryKeyCallBacks, |
&kCFTypeDictionaryValueCallBacks); |
} |
// example helper function to extract a time from our dictionary |
static int64_t GetFrameDisplayTimeFromDictionary(CFDictionaryRef inFrameInfoDictionary) |
{ |
CFNumberRef timeNumber = NULL; |
int64_t outValue = 0; |
if (NULL == inFrameInfoDictionary) return 0; |
timeNumber = CFDictionaryGetValue(inFrameInfoDictionary, CFSTR("MyFrameDisplayTimeKey")); |
if (timeNumber) CFNumberGetValue(timeNumber, kCFNumberSInt64Type, &outValue); |
return outValue; |
} |
Creating a Decoder
Listing 1 illustrates the basic steps required to create a decoder.
The function requires parameters describing the compressed source media bitstream (dimensions, format type and decoder configuration) to be passed in by the caller. These parameters are used to create the decoder configuration dictionary. An optional pixel format attribute dictionary is also created requesting '2vuy' as the format of the returned image buffers. This dictionary can be NULL
if the caller has no preference regarding the pixel format of the returned image buffers.
If decoder creation was successful, a VDADecoder
object is returned to the caller via decoderOut
.
Listing 1
OSStatus CreateDecoder(SInt32 inHeight, SInt32 inWidth, |
OSType inSourceFormat, CFDataRef inAVCCData, |
VDADecoder *decoderOut) |
{ |
OSStatus status; |
CFMutableDictionaryRef decoderConfiguration = NULL; |
CFMutableDictionaryRef destinationImageBufferAttributes = NULL; |
CFDictionaryRef emptyDictionary; |
CFNumberRef height = NULL; |
CFNumberRef width= NULL; |
CFNumberRef sourceFormat = NULL; |
CFNumberRef pixelFormat = NULL; |
// source must be H.264 |
if (inSourceFormat != 'avc1') { |
fprintf(stderr, "Source format is not H.264!\n"); |
return paramErr; |
} |
// the avcC data chunk from the bitstream must be present |
if (inAVCCData == NULL) { |
fprintf(stderr, "avc1 decoder configuration data cannot be NULL!\n"); |
return paramErr; |
} |
// create a CFDictionary describing the source material for decoder configuration |
decoderConfiguration = CFDictionaryCreateMutable(kCFAllocatorDefault, |
4, |
&kCFTypeDictionaryKeyCallBacks, |
&kCFTypeDictionaryValueCallBacks); |
height = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &inHeight); |
width = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &inWidth); |
sourceFormat = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &inSourceFormat); |
CFDictionarySetValue(decoderConfiguration, kVDADecoderConfiguration_Height, height); |
CFDictionarySetValue(decoderConfiguration, kVDADecoderConfiguration_Width, width); |
CFDictionarySetValue(decoderConfiguration, kVDADecoderConfiguration_SourceFormat, sourceFormat); |
CFDictionarySetValue(decoderConfiguration, kVDADecoderConfiguration_avcCData, inAVCCData); |
// create a CFDictionary describing the wanted destination image buffer |
destinationImageBufferAttributes = CFDictionaryCreateMutable(kCFAllocatorDefault, |
2, |
&kCFTypeDictionaryKeyCallBacks, |
&kCFTypeDictionaryValueCallBacks); |
OSType cvPixelFormatType = kCVPixelFormatType_422YpCbCr8; |
pixelFormat = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &cvPixelFormatType); |
emptyDictionary = CFDictionaryCreate(kCFAllocatorDefault, // our empty IOSurface properties dictionary |
NULL, |
NULL, |
0, |
&kCFTypeDictionaryKeyCallBacks, |
&kCFTypeDictionaryValueCallBacks); |
CFDictionarySetValue(destinationImageBufferAttributes, kCVPixelBufferPixelFormatTypeKey, pixelFormat); |
CFDictionarySetValue(destinationImageBufferAttributes, |
kCVPixelBufferIOSurfacePropertiesKey, |
emptyDictionary); |
// create the hardware decoder object |
status = VDADecoderCreate(decoderConfiguration, |
destinationImageBufferAttributes, |
(VDADecoderOutputCallback*)myDecoderOutputCallback, |
(void *)myUserData, |
decoderOut); |
if (kVDADecoderNoErr != status) { |
fprintf(stderr, "VDADecoderCreate failed. err: %d\n", status); |
} |
if (decoderConfiguration) CFRelease(decoderConfiguration); |
if (destinationImageBufferAttributes) CFRelease(destinationImageBufferAttributes); |
if (emptyDictionary) CFRelease(emptyDictionary); |
return status; |
} |
Output Callback
Listing 2 demonstrates a hypothetical output callback.
Clients should not perform any heavyweight tasks directly in the callback or call into any frameworks which may block for an extended period of time. If the intention is to display the frames, note that the image buffers may not be in display order. Ideally, frames returned in the output callback would be enqueued for processing or display on another thread.
Listing 2
void myDecoderOutputCallback(void *decompressionOutputRefCon, |
CFDictionaryRef frameInfo, |
OSStatus status, |
uint32_t infoFlags, |
CVImageBufferRef imageBuffer) |
{ |
MyUserDataPtr myUserData = (MyUserDataPtr)decompressionOutputRefCon; |
myDisplayFramePtr newFrame = NULL; |
myDisplayFramePtr queueWalker = myUserData->displayQueue; |
if (NULL == imageBuffer) { |
printf("myDecoderOutputCallback - NULL image buffer!\n"); |
if (kVDADecodeInfo_FrameDropped & infoFlags) { |
printf("myDecoderOutputCallback - frame dropped!\n"); |
} |
return; |
} |
if ('2vuy' != CVPixelBufferGetPixelFormatType(imageBuffer)) { |
printf("myDecoderOutputCallback - image buffer format not '2vuy'!\n"); |
return; |
} |
// allocate a new frame and populate it with some information |
// this pointer to a myDisplayFrame type keeps track of the newest decompressed frame |
// and is then inserted into a linked list of frame pointers depending on the display time |
// parsed out of the bitstream and stored in the frameInfo dictionary by the client |
newFrame = calloc(sizeof(myDisplayFrame), 1); |
newFrame->frame = CVBufferRetain(imageBuffer); |
newFrame->frameDisplayTime = GetFrameDisplayTimeFromDictionary(frameInfo); |
// since the frames we get may be in decode order rather than presentation order |
// our hypothetical callback places them in a queue of frames which will |
// hold them in display order for display on another thread |
pthread_mutex_lock(&myUserData->queueMutex); |
if (!queueWalker || (newFrame->frameDisplayTime < queueWalker->frameDisplayTime)) { |
// we have an empty queue, or this frame earlier than the current queue head |
newFrame->nextFrame = queueWalker; |
myUserData->displayQueue = newFrame; |
} else { |
// walk the queue and insert this frame where it belongs in display order |
Boolean frameInserted = false; |
myDisplayFramePtr nextFrame = NULL; |
while (!frameInserted) { |
nextFrame = queueWalker->nextFrame; |
if (!nextFrame || (newFrame->frameDisplayTime < nextFrame->frameDisplayTime)) { |
// if the next frame is the tail of the queue, or our new frame is ealier |
newFrame->nextFrame = nextFrame; |
queueWalker->nextFrame = newFrame; |
frameInserted = true; |
} |
queueWalker = nextFrame; |
} |
} |
myUserData->queueDepth++; |
pthread_mutex_unlock(&myUserData->queueMutex); |
} |
Decoding A Frame
Listing 3 demonstrates the use of VDADecoderDecode
to decompress a single frame of H.264 video.
The function takes a CFDataRef
containing the compressed video frame and a frame time (provided by the client from the bitstream) which is packaged in a frame info dictionary. The frame info dictionary is passed along to the output callback for this frame when decompressed and may contain any number of custom defined properties supplied by the client as key/value pairs.
Listing 3
OSStatus DecodeAFrame(VDADecoder inDecoder, CFDataRef inCompressedFrame, int64_t inFrameDisplayTime) |
{ |
CFDictionaryRef frameInfo = NULL; |
OSStatus status = kVDADecoderNoErr; |
// create a dictionary containg some information about the frame being decoded |
// in this case, we pass in the display time aquired from the stream |
frameInfo = MakeDictionaryWithDisplayTime(inFrameDisplayTime); |
// ask the hardware to decode our frame, frameInfo will be retained and pased back to us |
// in the output callback for this frame |
status = VDADecoderDecode(inDecoder, 0, inCompressedFrame, frameInfo); |
if (kVDADecoderNoErr != status) { |
fprintf(stderr, "VDADecoderDecode failed. err: %d\n", status); |
} |
// the dictionary passed into decode is retained by the framework so |
// make sure to release it here |
CFRelease(frameInfo); |
return status; |
} |
Reference
For Mac model information see: About integrated video on Intel-based Macs
Document Revision History
Date | Notes |
---|---|
2015-05-28 | Editorial |
2011-05-24 | Editorial |
2011-04-27 | Editorial |
2010-04-19 | Added a note describing where to get the framework to link against. |
2010-03-29 | New document that describes the Video Decode Acceleration Framework APIs available on Mac OS v10.6.3 and later with certain Mac models. |
Copyright © 2015 Apple Inc. All Rights Reserved. Terms of Use | Privacy Policy | Updated: 2015-05-28