Technical Note TN2112

Using the 3DMixer Audio Unit

This technote illustrates the usage of Core Audio's 3DMixer Audio Unit version 2.0.

The 3DMixer Audio Unit can mix audio from several different sources and then localize the sound in 3D space. This is very useful in games and in many other interactive applications that need to manage several audio sources simply and efficiently. This technote is intended for present and new users of the v2.0 3DMixer Audio Unit.

The v2.0 3DMixer Audio Unit has the ability to localize sound in 3-dimensional space, that is, position multiple sound sources by distance, elevation, and azimuth around a listener. The 3DMixer can render to multiple channel layouts (discussed below in the Speaker Configuration section). In addition, the quality of sound rendered during playback can be adjusted to meet CPU performance demands.

Creating an AUGraph for the 3DMixer
Setting the Bus count
Using the Internal Reverb
Speaker Configurations
Localizing sound in 3D space
Spatialization Rendering Algorithims
Setting the volume
Bus Stream Formats and the FormatConverterAU
Connecting, Initializing & Starting the AUGraph
Disposing of the AUGraph
Conclusion
References and Notes
Document Revision History

Creating an AUGraph for the 3DMixer

Figure 1  The 3D Mixer

3DMixers are typically inserted into an Audio Unit Graph (AUGraph) to help manage data flow. An AUGraph is a high-level representation of a set of AudioUnits, along with the connections between them. You can use the AUGraph APIs to construct arbitrary signal paths through which audio may be processed, i.e., a modular routing system. The APIs deal with large numbers of AudioUnits and their relationships.

The head of a graph is always an output unit, which may save the processed audio stream to disk, into memory, or as sound out. Starting a graph entails 'pulling' on the head audio unit (where data will exit the graph), which will, in turn, pull data from the connected audio units in the graph. For this technote, we will use default output audio unit as the head of the graph. The default output unit will represent the output device (speakers, in most cases) selected by the user in Audio MIDI Setup application or the System Prefrences Output Panel. For more information about using Audio Unit Graphs, please read Using the Audio Toolbox.

To begin setting up our graph, we must add nodes for the 3DMixer and a Default Output AudioUnit with the method AUGraphNewNode(). After creating the nodes, we can open the graph and obtain the AudioUnit instances from the nodes within the graph by using the method AUGraphGetNodeInfo().

Listing 1  Creating an AUGraph and adding nodes

void MyCreateAUGraph( AUGraph  mAUGraph,AudioUnit mOutputUnit,
 AudioUnit mMixerUnit)
{
    AUNode mMixerNode, mOutputNode;

    NewAUGraph(&mAUGraph); 

    ComponentDescription    mixerCD;

    mixerCD.componentFlags = 0; 
    mixerCD.componentFlagsMask = 0; 
    mixerCD.componentType = kAudioUnitType_Mixer; 
    mixerCD.componentSubType = kAudioUnitSubType_3DMixer; 
    mixerCD.componentManufacturer = kAudioUnitManufacturer_Apple; 

    AUGraphNewNode (mAUGraph, &mixerCD, 0, NULL, &mMixerNode); 

    ComponentDescription    cd;
    cd.componentFlags = 0; 
    cd.componentFlagsMask = 0; 

    cd.componentType = kAudioUnitType_Output; 
    cd.componentSubType = kAudioUnitSubType_DefaultOutput; 
    cd.componentManufacturer = kAudioUnitManufacturer_Apple; 

    AUGraphNewNode (mAUGraph, &cd, 0, NULL, &mOutputNode);

      //Open the Graph
     AUGraphOpen (mAUGraph);

     //Get Audio Units from graph
    AUGraphGetNodeInfo (mAUGraph, mMixerNode, 0, 0, 0, &mMixerUnit);
    AUGraphGetNodeInfo (mAUGraph, mOutputNode, 0, 0, 0, &mOutputUnit);

}

Now that the graph has been created we must setup the 3DMixer appprioprately before initializing and starting the graph.

Setting the Bus count

The number of inputs (or busses) of the 3Dmixer is a settable property. By default, the 3DMixer has 64 busses. Since busses incur memory usage, the bus count should not be set higher than necessary for the application's needs. Setting this property should be done prior to initializing the mixer unit.

Listing 2  Setting the mixer bus count

OSStatus    SetMixerBusCount (UInt32    inBusCount)
{

        UInt32  size        = sizeof(UInt32);
        UInt32  busCount    = inBusCount; 

        return (AudioUnitSetProperty ( mMixerUnit, 
                            kAudioUnitProperty_BusCount,
                            kAudioUnitScope_Input, 
                            0, 
                            &busCount,
                            size));

}

Using the Internal Reverb

The 3Dmixer has a built-in reverb. Once turned on, reverb is applied on a per input bus basis (as described below). Reverb is off by default and can be turned on with the AudioUnitSetProperty() method and the kAudioUnitProperty_UsesInternalReverb property, before initializing the graph.

Listing 3  Enabling the internal reverb

 UInt32        reverbSetting    = 1 // turn it on;
    result = AudioUnitSetProperty(mMixerUnit, 
                          kAudioUnitProperty_UsesInternalReverb, 
                          kAudioUnitScope_Global, 
                          0,
                          &reverbSetting, 
                          sizeof(reverbSetting));

Now that reverb has been turned ON for the 3DMixer, reverb can be enabled per bus by setting the k3DMixerRenderingFlags_DistanceDiffusion flag. Use the method AudioUnitSetProperty() and the kAudioUnitProperty_3DMixerRenderingFlags property to set the flag. This property can be set at anytime.

Listing 4  Turning reverb on for a bus

 UInt32    render_flags_3d;
    UInt32    outSize = sizeof(render_flags_3d);

    // get the current render flags for this bus
    result = AudioUnitGetProperty (mMixerUnit, 
            kAudioUnitProperty_3DMixerRenderingFlags, 
            kAudioUnitScope_Input,
            busIndex, 
            &render_flags_3d, 
            &outSize);

    // turn on this render flag and then set the bus
    render_flags_3d |= k3DMixerRenderingFlags_DistanceDiffusion;
    result = AudioUnitSetProperty( mMixerUnit,
             kAudioUnitProperty_3DMixerRenderingFlags,
             kAudioUnitScope_Input, busIndex, 
             &render_flags_3d, 
             sizeof(render_flags_3d));

Speaker Configurations

The v2.0 3DMixer can render audio to stereo, quad, and 5.0 channel layouts. Although the v2.0 3DMixer appropriately sets its channel layout when the output stream format is set, it is recommended that the channel layout be explicitly set. Therefore no changes are required to your code, should a future version of the mixer be released that supports multiple channel layouts that use the same number of channels (i.e. 5.1 and 6.0). When explicitly setting the channel layout be sure to pick a layout that matches the channel count of the stream format or an error will be returned (setting the channel layout to quad for example, if the stream format has 5 channels will return an error). In order to take full advantage of the mixer's ability to render to multiple channel layouts, it is necessary to determine what the user has set for the Speaker Configuration in the Audio MIDI Setup application (see example below) by querying the current device for it's output channel layout.

By querying the output unit of the graph, the following code determines an appropriate channel count to be used when setting the stream format (output scope) of the mixer.

Listing 5  Determining the correct number of channels to render to

UInt32 GetDesiredRenderChannelCount ()
{
    OSStatus result = noErr;

    // get the HAL device id form the output AU 
    AudioDeviceID    deviceID;
    UInt32    returnValue = 2; // return stereo by default

    UInt32 outSize =  sizeof(deviceID);

    //get the current device
    AudioUnitGetProperty(mOutputUnit, 
                        kAudioOutputUnitProperty_CurrentDevice,
                        kAudioUnitScope_Output, 
                        1,
                        &deviceID, 
                        &outSize);

    //Get the users speaker configuration
    result = AudioDeviceGetPropertyInfo(deviceID, 
                        0, 
                        false, 
                        kAudioDevicePropertyPreferredChannelLayout, 
                        &outSize,
                        NULL);

    if (result != noErr)
        return (returnValue); // return default (stereo) 

    AudioChannelLayout    *layout = NULL;
    layout = (AudioChannelLayout *) calloc(1, outSize);
    if (layout != NULL)
    {
        result = AudioDeviceGetProperty(deviceID, 
                       0, 
                       false, 
                       kAudioDevicePropertyPreferredChannelLayout, 
                       &outSize,
                       layout);
        if (layout->mChannelLayoutTag 
                       == kAudioChannelLayoutTag_UseChannelDescriptions)
        {
        // no channel layout tag is returned, 
                //so walk through the channel descriptions and count
                // the channels that are associated with a speaker
        if (layout->mNumberChannelDescriptions == 2)
        { 
                returnValue = 2; // there is no channel info for stereo
        }
        else
        {
            returnValue = 0;
            for (UInt32 i = 0; i < layout->mNumberChannelDescriptions; i++)
            {
                if (layout->mChannelDescriptions[i].mChannelLabel !=
                                             kAudioChannelLabel_Unknown) 
                    returnValue++;
                }
            }
        }
        else
        {
            switch (layout->mChannelLayoutTag)
            {
                case kAudioChannelLayoutTag_AudioUnit_5_0:
                case kAudioChannelLayoutTag_AudioUnit_5_1:
                case kAudioChannelLayoutTag_AudioUnit_6:
                    returnValue = 5;
                    break;
                case kAudioChannelLayoutTag_AudioUnit_4:
                    returnValue = 4;
                default:
                    returnValue = 2;
            }
        }

        free(layout);
    }
  return returnValue;
}

Listing 6  Configuring the AUGraph For Channel Layout

void ConfigureGraphForChannelLayout()
{
    OSStatus    result = noErr;

    // get the channel count that should be set
        // for the mixer's output stream format
    mCurrentMixerChannelCount = GetDesiredRenderChannelCount();

    // set the stream format
    CAStreamBasicDescription format; //CoreAudio SDK class
    UInt32 outSize = sizeof(format);
    result = AudioUnitGetProperty(mOutputUnit, 
                               kAudioUnitProperty_StreamFormat,
                               kAudioUnitScope_Output,
                               0, 
                               &format,
                               &outSize);

        // not interleaved
    format.SetCanonical (mCurrentMixerChannelCount, false); 
    format.mSampleRate = mMixerOutputRate;
    outSize = sizeof(format);
    result = AudioUnitSetProperty (mOutputUnit,
                               kAudioUnitProperty_StreamFormat, 
                               kAudioUnitScope_Input, 
                               0, 
                               &format, 
                               outSize);

    result = AudioUnitSetProperty (mMixerUnit, 
                               kAudioUnitProperty_StreamFormat, 
                               kAudioUnitScope_Output, 
                               0, 
                               &format, 
                               outSize);

    return;
}

Localizing sound in 3D space

Setting the distance, elevation and azimuth of a bus causes the listener to perceive the audio from a specific point in a 3-dimensional space.

The pitch and gain of the audio sources can be changed per bus within the 3DMixer. The pitch can be changed by modifying the value of the parameter k3DMixerParam_PlaybackRate. The gain can be changed by modifying the value of the parameter k3DMixerParam_Gain (example shown in "Setting the Volume").

Table 1  3-Dimensional Parameters (Location)

Direction

Unit

Range

Azimuth

Degrees

-180 to 180

Elevation

Degrees

-90 to 90

Distance

Meters

0 to 10000

Table 2  3-Dimensional Parameters ( Volume and Pitch)

Direction

Unit

Range

Gain

dB

-120 to 20

Playback Rate (Pitch)

rate

0.5 to 4.0

Listing 7  Setting the distance and azimuth

void    SetObjectCoordinates(UInt32     inMixerInputBus, Float32,
inAzimuth, Float32 inDistance)
{
    AudioUnitSetParameter(mMixerUnit, 
                                            k3DMixerParam_Azimuth, 
                                            kAudioUnitScope_Input, 
                                            inMixerInputBus, 
                                            inAzimuth, 
                                            0);

    AudioUnitSetParameter(mMixerUnit,
                                            k3DMixerParam_Distance, 
                                            kAudioUnitScope_Input,
                                            inMixerInputBus, 
                                            inDistance, 
                                            0);
}

Starting with the v2.0 3DMixer, it is possible to have the mixer do all the work necessary to clamp the bus volume based on specified ReferenceDistance, MaximumDistance and Attenuation settings. This feature allows an application to specify how close/far away an object in 3D space can actually get to/from the listener. The following method demonstrates how to properly set/get a mixer bus's DistanceParams with the AudioUnitSetProperty() method and the kAudioUnitProperty_3DMixerDistanceParams property.

Listing 8  Mixer Distance Parameters struct defined in AudioUnitProperties.h

typedef struct MixerDistanceParams {     Float32            mReferenceDistance;     Float32            mMaxDistance;     Float32            mMaxAttenuation; // in decibels } MixerDistanceParams;
Figure 2  Distance Parameters Graph

Listing 9  Setting the Distance Parameters for a bus

void    SetDistanceParamsForBus (UInt32     inMixerInputBus)
{ 
         MixerDistanceParams    distanceParams;

    // Get the desired minimum distance an object 
    // can get to the listener (float value in meters)
    // The mixer will play audio on the bus with 0db 
    // of attenuation (no attenuation) for all distance coordinates
    // between 0.0 and this reference distance. This 
    //value must be less than the maximum distance setting.
    distanceParams.mReferenceDistance = GetMyMinimumDistance();

    // Get the desired maximum distance an object 
    // can get from the listener (float value in meters)
    // The mixer will stop attenuating audio on the bus 
    // for all distance coordinates at this maximum 
    // distance or greater. This value must be greater 
    // than the reference distance setting.
    distanceParams.mMaxDistance = GetMyMaximumDistance();

    // Get desired attenuation (in db) for an object when it's 
    // coordinates are at or beyond the maximum distance 
    // (positive float value in dB).
    // The mixer will not attenuate the audio further than 
    // this db setting when the distance coordinates are beyond the 
    // maximum distance setting. 

    distanceParams.mMaxAttenuation = GetMyMinimumAttenuation();

    OSStatus result = AudioUnitSetProperty(mMixerUnit, 
                                kAudioUnitProperty_3DMixerDistanceParams,
                                kAudioUnitScope_Input,
                                inMixerInputBus, 
                                &distanceParams, 
                                sizeof(distanceParams));


}

Spatialization Rendering Algorithims

Spatialization settings are on a per bus (mixer input) basis and set with AudioUnitSetProperty() method and the kAudioUnitProperty_SpatializationAlgorithm constant. The type of the spatialization rendering used is determined by factors such as source audio channels (mono/stereo), output hardware channels (stereo, multichannel), and cpu capability. Setting a bus spatialization algorithm may also change some of the bus rendering flags (kAudioUnitProperty_3DMixerRenderingFlags) so it is recommended to reset any desired render flags after setting the spatialization algorithm.

Table 3  Spatailization Algorithims appropriate for Stereo Output (from AudioUnitProperies.h)

Algorithims

Description

Output

kSpatializationAlgorithm_StereoPassThrough

StereoPassThrough should be used when no localization is desired for stereo source data. Setting this algorithm tells the mixer to take stereo input and pass it directly to channels 1 & 2 (Front L/R if rendering is for multichannel) without localization.

Stereo , Multi-Channel

kSpatializationAlgorithm_EqualPowerPanning

EqualPowerPanning merely pans the data of the mixer bus into a stereo field. This algorithm is analogous to the pan knob found on a mixing board channel strip. When this algorithm is used while the mixer is rendering to multichannel hardware audio data will only be rendered to channels 1 & 2 (Front L/R).

Stereo

kSpatializationAlgorithm_HRTF

HRTF (Head Related Transfer Function) is a high quality algorithm using filtering to emulate 3 dimensional space in headphones or stereo speakers. If turned on while rendering to multi channel hardware, HRTF will render the input data only to channels 1 & 2 (Front L/R). HRTF is a cpu intensive algorithm.

Stereo

kSpatializationAlgorithm_SphericalHead

Like HRTF, the SpericalHead algorithm is designed to emulate 3 dimensional space in headphones or stereo speakers. Using simplified algorithms, SpericalHead provides a lesser quality rendering experience than HRTF and is slightly less cpu intensive.

Stereo

kSpatializationAlgorithm_VectorBasedPanning

VectorBasedPanning pans data from the bus, based on it's location in 3 dimensional space, between two channels of the multichannel rendering (i.e. between channels 3 & 4 for a quad rendering if the location of the data was directly behind the listener). When this algorithm is used while the mixer is rendering to stereo hardware, it effectively becomes EqualPowerPanning (see EqualPowerPanning for more info.) This algorithm does not pan moving sources as smoothly as the Sound Field (kSpatializationAlgorithm_SoundField) algorithm and is better suited for localizing fixed sources.

Multi-Channel

kSpatializationAlgorithm_SoundField (Ambisonics)

SoundField is designed for rendering to multi channel hardware. The mixer takes data being rendered with SoundField and distributes it amongst all the output channels with a weighting toward the location in which the sound derives. It is very effective for ambient sounds, which may derive from a specific location in space, yet should be heard through the listener's entire space.

Multi-Channel

Listing 10  Setting the 3DMixer to a Spatialization Algorithims

SetSpatializationAlgorithm( UInt32 BusNumber)
{
          UInt32 spatialSetting = GetMySpatializationSetting(); 

          result = AudioUnitSetProperty(mMixerUnit, 
                             kAudioUnitProperty_SpatializationAlgorithm, 
                             kAudioUnitScope_Input,
                             BusNumber,
                             &spatialSetting,
                             sizeof(spatialSetting));
}

Setting the volume

The volume can be explicitly set for each bus on the input scope of the 3DMixer, providing a gain before the sound is mixed. Use the k3DMixerParam_Gain constant and AudioUnitSetParameter() method to set the volume of each mixer bus (input scope). The parameter is a float value representing dB (see below).

Listing 11  Setting the volume on input

// busGain represents a range of 0.0 to 1.0 (full volume) 
SetInputBusGain(UInt32 mCurrentPlayBus, Float32 busGain)
{
   Float32    db = 20.0 * log10(busGain); // convert to db
   if (db < -120.0)
      db = -120.0; // clamp minimum audible level at -120db

   AudioUnitSetParameter(mMixerUnit, 
                                       k3DMixerParam_Gain,
                                       kAudioUnitScope_Input,
                                       mCurrentPlayBus, 
                                       db, 
                                       0);
}

Bus Stream Formats and the FormatConverterAU

If a FormatConverterAU is supplying the data to the mixer bus, then the stream format on the input scope of the mixer is automatically set when the nodes are connected. However, if the data is being supplied to the mixer directly in the render proc of the bus, it is necessary to properly set the stream format on the input scope of a mixer's bus before 'pulling' for data.

Connecting, Initializing & Starting the AUGraph

Setup is now complete and we can connect the 3DMixer to the default output audio unit. When all connections are made and nodes are setup up correctly, you must initialize the AUGraph. The graph is now complete and the 3Dmixer is ready for use. A call to AUGraphStart will start processing the audio data.

Listing 12  Connecting, Initializing & Starting the AUGraph

 AUGraphConnectNodeInput (mAUGraph, mMixerNode, 0, mOutputNode, 0);         AUGraphInitialize (mAUGraph);         AUGraphStart(mAUGraph);

Disposing of the AUGraph

The method AUGraphStop will stop all rendering throughout a graph. DisposeAUGraph will deallocate the graph.

Listing 13  Stopping and disposing the graph

 AUGraphStop (mAUGraph);          DisposeAUGraph (mAUGraph);         mAUGraph = 0;

Conclusion

CoreAudio's 3DMixer is very a useful audio tool. The mixers ability to handle audio from multiple sources and localize the sounds in 3D space is a necessity in games and can be amazing in countless other applications.

References and Notes

Core Audio Preliminary doc



Document Revision History


DateNotes
2004-06-14

Updated Reference Library Location

2004-05-26

New document that discusses how to use Core Audio's 3DMixer version 2.0