Passing Non-Media Data

Media packetizers and packet reassemblers are generally concerned with media sample data, such as video frames or audio samples. QuickTime movies commonly contain track-level information, such as a transformation matrix, or audio volume, that is applied to the sample data but is not embedded in it.

Packetizers and reassemblers can optionally support the passing of this type of data. If your packetizer supports passing of a particular type of information, such as audio volume or transformation matrix, it should indicate this in its public resource (as described in The 'pcki' Public Resource).

In the course of normal operations, a media packetizer may receive a SetInfo command that specifies this type of information. A packetizer that can send a transformation matrix, for example, could receive such a call whenever the matrix changes. It is up to the packetizer to package this information in a way that will be recognized by the reassembler and to send it along.

When a reassembler receives this type of information, it signals the fact by calling RTPRssmSendStreamHandlerChanged. It may then receive a series of GetInfo calls to determine what information it has. It passes the information in response to these calls, and QuickTime does the rest.

The 'pcki' Public Resource

A packetizer must provide a public resource of type 'pcki'. The public resource contains information about the capabilities of a given packetizer. This information lists the media types and compression formats the packetizer can work with. It also lists the track characteristics the packetizer can work with, such as layers or transformation matrices. In addition, it provides information about the packetizer’s performance characteristics, such as its speed or ability to recover from packet loss.

QuickTime selects the packetizer best suited to the media, compression format, and track characteristics. If there are multiple packetizers that can work with a given track, performance is considered.

For example, one packetizer might be able to pack H.261-compressed video, but be unable to handle any transformation matrix other than identity. Another packetizer might also do H.261 and be able to handle any kind of transformation matrix. If a matrix was being used to scale and translate the media, QuickTime would select the more versatile packetizer. If there were no matrix applied to the track, QuickTime might select the faster packetizer for a live transmission, and the packetizer that dealt best with loss when creating a hint track.

A movie that contains streaming tracks can be played from a local disk or served via HTTP. You can create a client movie with a streaming track that contains the URL of a server movie, then embed the client movie in a website or distribute it on a CD-ROM. When the movie is played, the user’s computer will establish a network connection to the specified server for each streaming track.

The format of the public resource is defined in QTStreamingComponents.r. It is shown in Listing 3-1.

Listing 3-1  Format of the 'pcki' resource

type 'pcki' {

    hex longintmediaType;

    hex longintdataFormat;

    hex longintvendor;

    hex longintcapabilityFlags;

    byte    canPackMatrixType;

    byte = 0;

    byte = 0;

    byte = 0;

    longint = $$CountOf(characteristicArray); /* Array size*/

    array characteristicArray {

        hex longinttag;

        hex longint value;

    };  

    hex longintpayloadFlags;

    byte    payloadID;      /* if static payload */

    byte = 0;

    byte = 0;

    byte = 0;

    cstring;

};

The first three fields describe the media type, data format (compression format), and the manufacturer. The media type and data format are matched against the given track and media information when deciding whether to use a given packetizer. If your packetizer supports multiple compression formats, set dataFormat to 0. If it supports multiple media types, set mediaType to 0. The vendor field is for informational purposes only.

The capabilityFlags field can contain a combination of these flag constants:

enum {

    kMediaPacketizerCanPackEditRate = 1 << 0,

    kMediaPacketizerCanPackLayer = 1 << 1,

    kMediaPacketizerCanPackVolume = 1 << 2,

    kMediaPacketizerCanPackBalance = 1 << 3,

    kMediaPacketizerCanPackGraphicsMode = 1 << 4,

    kMediaPacketizerCanPackEmptyEdit = 1 << 5

};

These flags describe a packetizer’s ability to deal with track characteristics. By indicating a capability, a packetizer says that it can transmit track information (such as a matrix) to the reassembler on the client side, so that it can be used in the client movie. The specific flags have these meanings:

The canPackMatrixType flag is one of:

canPackIdentityMatrixType 0x00

canPackTranslateMatrixType 0x01

canPackScaleMatrixType 0x02

canPackScaleTranslateMatrixType 0x03

canPackLinearMatrixType 0x04

canPackLinearTranslateMatrixType 0x05

canPackPerspectiveMatrixType 0x06

Note that these are the same values as those defined in ImageCompression.h as return values for GetMatrixType. The value indicates you can communicate the specified matrix type.

There are two performance characteristics currently defined:

#define kMediaPacketizerSpeedTag          'sped'    /* 0-255, 255 is fastest */

#define kMediaPacketizerLossRecoveryTag   'loss'    /* 0-255, 0 can't handle

                                    any loss, 128 can handle 50% packet loss */  

The speed tag is relative to other packetizers for the same media type and compression format. A value of 128 is a reasonable default.

The payloadFlags field is set to either kRTPMPPayloadTypeDynamicFlag for a dynamic payload type or kRTPMPPayloadTypeStaticFlag for a static payload type.

The payload ID field of the 'pcki' resource is set to the IETF-defined RTP payload value if a static payload type is used.

The C string contains the RTP payload type text if a dynamic payload type is used.

A code example of a resource for a packetizer that supports the 'OVAL' video compression format is this:

resource 'thnr' (128) {

    {

        'pcki', 1, 0,

        'pcki', 128, cmpResourceNoFlags,

    };

};      

resource 'pcki' (128) {

    'vide',         // media type

    'OVAL',         // data format type

    'ABCD',         // manufacturer type

    kMediaPacketizerCanPackEditRate,

    canPackIdentityMatrixType,

    {

        kMediaPacketizerSpeedTag, 128,

        kMediaPacketizerLossRecoveryTag, 50

    },

    kRTPMPPayloadTypeDynamicFlag,

    0,

    "OVAL-49545"

};

This packetizer is of medium speed, can handle some loss, can pack any arbitrary play rate, but can’t pack any non-identity matrix. Its RTP payload type is a dynamic identifier, identified by the string OVAL-49545.

Media Packetizer Functions

QuickTime Streaming calls a media packetizer during the course of hinting a movie or during live transmission. The packetizer is passed sample data, which it breaks into packets and passes to a packet builder. The packet builder may then transmit the packets over RTP or use them to create a hint track. The media data is presented to the packetizer in the same format for a live transmission or for hinting, and the packetizer produces the same output in both cases.

The packetizer component must implement several functions, and must also provide a public component resource that describes the type of media, compression, and track characteristics that the packetizer supports. This resource also provides information on the packetizer’s relative speed and the format’s ability to handle loss.

The following functions can be implemented by media packetizer components. Most of these functions must be implemented in your component, but some are optional.

• RTPMPInitialize

• RTPMPPreflightMedia

• RTPMPIdle

• RTPMPSetSampleData

• RTPMPReset

• RTPMPSetInfo

• RTPMPGetInfo

• RTPMPSetTimeScale

• RTPMPGetTimeScale

• RTPMPSetTimeBase

• RTPMPGetTimeBase

• RTPMPHasCharacteristic

• RTPMPSetPacketBuilder

• RTPMPGetPacketBuilder

• RTPMPSetMediaType

• RTPMPGetMediaType

• RTPMPSetMaxPacketSize

• RTPMPGetMaxPacketSize

• RTPMPSetMaxPacketDuration

• RTPMPGetMaxPacketDuration

• RTPMPDoUserDialog

• RTPMPSetSettingsFromAtomContainerAtAtom

• RTPMPGetSettingsIntoAtomContainerAtAtom

• RTPMPGetSettingsAsText

Sequence of Events

When QuickTime is asked to provide a packetizer, it selects the packetizer based on the media type, data format, and other characteristics, such as whether a matrix transformation is in use. It selects the packetizer best able to handle the media and track characteristics by examining the packetizer’s public resource.

Once a packetizer has been selected, it is opened. It is then asked to preflight the media, to verify that it can actually packetize the desired media data. If the packetizer indicates that it can handle the media, it is initialized. The packetizer then receives a series of setup calls required to prepare it for operation. These calls deliver information such as the media time scale and the packet builder to use for output.

Once setup is complete, the packetizer receives a series of calls with sample data. If the packetizer can process the data immediately, it does so; otherwise it returns a flag that indicates it is still processing the data. In the latter case, the packetizer’s Idle function is called perodically until the packetizer has completed its processing. When it is ready, the packetizer creates packets by making calls to a packet builder component.

The packetizer is then called again with more sample data. This continues until all the media data has been packetized.

At any time in this process, the packetizer can be asked to return information, or to flush its input buffer, or to reset itself and prepare for a new sequence.

Packetizer Component Type and Subtype

Packetizers have a component type of kRTPMediaPacketizerType ('rtpm'). The subtype can be any four-character combination. Note, however, that all-lowercase types are reserved by Apple.

Media Preflight

Once QuickTime has selected and opened a packetizer component, it will call the packetizer’s RTPMPPreflightMedia function to verify that the packetizer can handle the specific media and sample description. A packetizer can reject media as a result of this call, even if the media fits the profile described in the packetizer’s public resource.

For example, the 'pcki' resource for the Qualcomm PureVoice audio packetizer indicates support for audio media and PureVoice compression, but the packetizer only supports 8 kHz sample rates over RTP. The RTPMPPreflightMedia call to this packetizer returns noErr for 8 kHz audio, and an error for any other sample rate.

The media preflight function is defined as follows:

pascal ComponentResult RTPMPPreflightMedia(RTPMediaPacketizer rtpm,

                OSType inMediaType,

                SampleDescriptionHandle inSampleDescription);

This call is made to make sure your packetizer can handle a given media type and sample description. Return noErr if you can support it. Return qtsUnsupportedFeatureErr if you cannot.

Initialization

If your packetizer returns noErr during the media preflight, it will be initialized before it is asked to handle any data. The initialization function is defined as follows:

pascal ComponentResult RTPMPInitialize(RTPMediaPacketizer rtpm,

                SInt32 inFlags);

The value of inFlags can be 0 or kRTPMPRealtimeModeFlag = 0x00000001.

The kRTPMPRealtimeModeFlag flag indicates that your packetizer is being used for live transmission, rather than for hinting. You might use this information if your packetizer has a tradeoff between speed and fidelity.

Setup and Information Functions

Once your packetizer is initialized, it will receive several calls to set the information it needs prior to packetizing data, such as the media timescale. It may also be called with requests to return the settings it has been given. The main setup functions are these:

• RTPMPSetTimeScale

• RTPMPGetTimeScale

• RTPMPSetPacketBuilder

• RTPMPGetPacketBuilder

• RTPMPSetMediaType

• RTPMPGetMediaType

• RTPMPSetMaxPacketSize

• RTPMPGetMaxPacketSize

• RTPMPSetMaxPacketDuration

• RTPMPGetMaxPacketDuration

The Set functions listed above are used to set the time scale, packet builder (which receives your output), media type, maximum packet size, and maximum packet duration. These functions will be called before you are asked to begin packetizing data, and will not be called after you have begun packetizing data.

The Get functions listed above can be called at any time. When your packetizer is called with one of these Get functions, return the data that was passed to you in the corresponding Set function.

• RTPMPSetTimeBase

• RTPMPGetTimeBase

The time base functions may be called for live transmission. The Set function sets the QuickTime time base that is in use. Your packetizer can query this time base to find out what time it is in the live stream. Your packetizer should not rely on receiving this call.

RTPMPHasCharacteristic may be called to determine whether your media packetizer has a particular characteristic, such as whether it supports a user settings dialog. Return qtsBadSelectorErr if your packetizer does not have the given characteristic.

ComponentResult RTPMPHasCharacteristic (

                     RTPMediaPacketizer rtpm,

                     OSType inSelector,

                     Boolean *outHasIt);

The inSelector parameter may have these values:

If your packetizer supports a user dialog or additional settings, you may also receive these dialog or settings calls as part of the set up process:

RTPMPDoUserDialog

RTPMPSetSettingsFromAtomContainerAtAtom

RTPMPGetSettingsIntoAtomContainerAtAtom

RTPMPGetSettingsAsText

• The RTPMPDoUserDialog function will invoke your packetizer’s modal dialog to obtain user settings.

• The RTPMPGetSettingsIntoAtomContainerAtAtom function expects you to return those settings into an atom container at the specified offset.

• The RTPMPSetSettingsFromAtomContainerAtAtom function is used to set the user settings programmatically, bypassing the user dialog.

• The RTPMPGetSettingsAsText function expects you to return your user settings as text.

In addition to the set up calls listed above, you may receive RTPMPSetInfo function calls. These are used to send a variety of information to your packetizer, as indicated by the selector. If you don’t support a given selector, return qtsBadSelectorErr.

The RTPMPSetInfo selectors can be:

kQTSSourceTrackIDInfo ('otid'),         /* UInt32* */

kQTSSourceLayerInfo ('olyr'),           /* UInt16* */

kQTSSourceLanguageInfo ('olng'),        /* UInt16* */

kQTSSourceTrackFlagsInfo ('otfl'),      /* SInt32* */

kQTSSourceDimensionsInfo ('odim'),      /* QTSDimensionParams* */

kQTSSourceVolumesInfo ('ovol'),         /* QTSVolumesParams* */

kQTSSourceMatrixInfo ('omat'),          /* MatrixRecord* */

kQTSSourceClipRectInfo ('oclp'),        /* Rect* */

kQTSSourceGraphicsModeInfo ('ogrm'),    /* QTSGraphicsModeParams* */

kQTSSourceScaleInfo ('oscl'),           /* Point* */

kQTSSourceBoundingRectInfo ('orct'),    /* Rect* */

kQTSSourceUserDataInfo ('oudt'),        /* UserData */

kQTSSourceInputMapInfo ('oimp'),        /* QTAtomContainer */

Your packetizer can be called with these selectors even if it indicated that it couldn’t handle the given characteristic in its 'pcki' resource. In that case, return qtsBadSelectorErr.

Your packetizer may receive the related RTPMPGetInfo function at any time:

pascal ComponentResult RTPMPGetInfo(RTPMediaPacketizer rtpm,

                OSType inSelector, void *ioParams);

Your packetizer is expected to return the requested information in ioParams. The type of ioParams is dependent on inSelector. If you don’t support the selector, return qtsBadSelectorErr. The selectors can be any of the selectors for RTPMPSetInfo or any of the additional selectors listed below:

/* info selectors -- get only */

kRTPMPPayloadTypeInfo ('rtpp'),                /* RTPMPPayloadTypeParams* */

kRTPMPRTPTimeScaleInfo ('rtpt'),               /* TimeScale* */

kRTPMPRequiredSampleDescriptionInfo  ('sdsc'), /* SampleDescriptionHandle* */

kRTPMPMinPayloadSize ('mins'),                 /* UInt32*, doesn't include

                                                  rtp header; default 0 */ 

kRTPMPMinPacketDuration ('mind'),              /* UInt3* in milliseconds;

                                                  default is no min */ 

kRTPMPSuggestedRepeatPktCountInfo  ('srpc'),   /* UInt32* */

kRTPMPSuggestedRepeatPktSpacingInfo ('srps'),  /* UInt32* in milliseconds */

kRTPMPMaxPartialSampleSizeInfo  ('mpss'),      /* UInt32* in bytes */

kRTPMPPreferredBufferDelayInfo  ('prbd')       /* UInt32* in milliseconds */

The kRTPMPPayloadTypeInfo selector requires you to fill out an RTPMPPayloadTypeParams structure:

struct RTPMPPayloadTypeParams {

    UInt32  flags;

    UInt32  payloadNumber;

    short   nameLength;

/* in: size of payloadName buffer (counting null terminator);

/* this will be reset to needed length and paramErr returned if too small */

    char *  payloadName;    /* caller must provide buffer */

};

typedef struct RTPMPPayloadTypeParams RTPMPPayloadTypeParams;

/* flags for RTPMPPayloadTypeParams */

enum {

    kRTPMPPayloadTypeStaticFlag= 0x00000001,

    kRTPMPPayloadTypeDynamicFlag = 0x00000002

};

If you have a dynamic RTP payload type, you need to copy the payload type string to the buffer pointed to by payloadName (a null-terminated string). When RTPMPGetInfo is called, RTPMPPayloadTypeParams will be the available size of the input buffer (specified by payloadName). If this size is too small for the payload identifier, set nameLength to the size of the buffer you need (including the null terminator) and return paramErr. This will cause QuickTime to reallocate the buffer and call your packetizer again.

Sample Processing Functions

Once setup is complete, QuickTime will begin calling your packetizer’s RTPMPSetSampleData function:

pascal ComponentResult RTPMPSetSampleData(RTPMediaPacketizer rtpm,

                const RTPMPSampleDataParams *inSampleData, SInt32 *outFlags);

This is where most of a typical packetizer’s work is done. Your packetizer is presented with a block of media sample data, you break it into packets according to your own algorithm, and you pass the results to the selected packet builder.

• If your packetizer works synchronously, you packetize the data and return 0 in outFlags. If you need more sample data to complete a packet, return kRTPMPWantsMoreDataFlag.

• If your packetizer works asynchronously, you set outFlags to kRTPMPStillProcessingData and your packetizer continues its work in the RTPMPIdle function, which will be called periodically. RTPMPIdle should also set outFlags to kRTPMPStillProcessingData if it still has work to do. It sets outFlags to 0 if all the sample data has been processed.

• If your packetizer works synchronously, RTPMPIdle always sets outFlags to 0.

Your media packetizer must call the release procedure when done with the media data.

Do the processing work in the RTPMPSetSampleData function if it does not take up too much CPU time; otherwise, do it in your idle function:

struct RTPMPSampleDataParams {

    UInt32                  version;

    UInt32                  timeStamp;

    UInt32                  duration;           /* 0 if not specified*/ 

    UInt32                  playOffset;         /* 0 if not specified*/ 

    Fixed                   playRate;

    SInt32                  flags;

    UInt32                  sampleDescSeed;

    Handle                  sampleDescription;

    RTPMPSampleRef          sampleRef;

    UInt32                  dataLength;

    const UInt8 *           data;

    RTPMPDataReleaseUPP     releaseProc;

    void *                  refCon;

};

typedef struct RTPMPSampleDataParamsRTPMPSampleDataParams;

/* flags for RTPMPSampleDataParams*/

enum {

    kRTPMPSyncSampleFlag= 0x00000001

};

If you can’t do all the processing of the sample data in response to the RTPMPSetSampleData function, sets outFlags to kRTPMPStillProcessingData and do the work in your RTPMPIdle function:

// do work here if you need to -- give up time periodically

// if you're doing time consuming operations

pascal ComponentResult RTPMPIdle(RTPMediaPacketizer rtpm,

                SInt32 inFlags, SInt32 *outFlags);

This call is made periodically if you set outFlags to kRTPMPStillProcessingData in your RTPMPSetSampleData routine. If you need more time, sets outFlags to kRTPMPStillProcessingData in RTPMPIdle as well. This will cause RTPMPIdle to be called again. When you are finished packetizing the sample data you were passed in the last RTPMPSetSampleData call, sets outFlags to 0.

If you do work in the RTPMPIdle function, the idle function needs to call the release procedure when you are done with the sample data.

Calling the Packet Builder

The actual work your packetizer does in the RTPMPSetSampleData or RTPMPIdle function consists of creating packets. This is done by making calls to a packet builder component. The packet builder functions are defined in QTStreamingComponents.h.

The first thing you will need to do is to begin a new packet group. To start a new group of packets, call RTPPBBeginPacketGroup. Packets in a single group have the same RTP time stamp (and go into the same hint sample, if the data is being used to create a hint track).

Next, you will want to add one or more packets to the group.

• Call RTPPBBeginPacket to begin a single network packet.

• Call RTPPBAddPacketLiteralData to add literal data to the packet, such as header information.

• Call RTPPBAddPacketSampleData to add a sample reference to the packet, such as the sample data specified in inSampleDataParams in a call to your packetizer’s RTPMPSetSampleData function.

You may want to packetize the same data repeatedly. There are utility functions to make this easier. Begin by passing in RTPPacketRepeatedDataRef to RTPPBAddPacketLiteralData or RTPPBAddPacketSampleData, depending on whether you want to repeat literal data or sample data. The packet builder will store a copy of the data and return a data reference. You then call RTPPBAddPacketRepeatedData with the data reference whenever you want to add copies of the data. Call RTPPBReleaseRepeatedData when you are done with the data. The packet builder will maintain a copy of the data until you release it.

End a packet by calling RTPPBEndPacket. End the group of packets by calling RTPPBEndPacketGroup.

Flush and Reset Routines

Your packetizer may be called with commands to flush its input buffer or reset for a new session at any time. This normally happens at the end of a sequence or when a transmission is interrupted.

The flush command is given when your packetizer needs to finish any pending work:

pascal ComponentResult RTPMPFlush(RTPMediaPacketizer rtpm,

                SInt32 inFlags, SInt32 *outFlags);

If your packetizer has not yet written all the data from the last RTPMPSetSampleData call to the selected packet builder, upon receiving the flush command it should write any pending data, flush its input buffer and set outFlags to 0. The inFlags value is currently always 0.

The reset command is given when your packetizer needs to stop packetizing, reset its state, and prepare for new orders:

pascal ComponentResult RTPMPReset(RTPMediaPacketizer rtpm,

                SInt32 inFlags);

The inFlags value is currently always 0.

Flush your input buffer. Do not send any buffered data, because in all probability there is no connection for you to send on. Reset your packetizer’s state; it should be the same as if it had just been opened and initialized.