Learn about using Audio Unit Extensions to provide your App with sophisticated audio manipulation and processing capabilities. Explore the architecture and fundamentals of an App that uses Audio Unit Extensions, and how your app can benefit from their power.
DOUG WYATT: Good morning.
I'm Doug Wyatt from the Core Audio team and I would
like to show you something new we have been working
on called Audio Unit Extensions.
This is a new technology in iOS 9 and OS X El Capitan.
About Audio Units: we've had had this technology
in our operating systems since the beginning OS X and iOS.
The operating system includes a great number
of built-in units ranging from I/O units and mixers,
a lot of different effects ranging to software sampler.
We use these internal Audio Units in many
of our higher-level APIs, for example,
the media playback stack.
But Audio Units are also a widely adopted third-party
plug-in format on OS X.
There are literally thousands of third-party Audio Units
in the market out there.
Now, Audio Unit extensions
for the first time bring us a full plug-in model
on both OS X and iOS.
It is built on top of the app extension technology,
which means if you are writing plug-ins you can package them
into apps, and those apps can be sold on the App Stores.
As part of this technology, we have modernized the API and
yet at the same time maintained compatibility,
and in this session I will go through details of this new API,
which we are calling the version 3 Audio Unit API.
It's based on an Objective-C class called AUAudioUnit that's
in the Audio Unit framework.
And as an Objective-C class, of course, it plays nicely
with Swift, as we will see.
In this session, we are also going to look at a number
of classes in the AVFoundation framework.
We have AV Audio Unit component manager
and AV Audio Unit component.
These are used to located the audio components on the system.
Those appear for the first time in iOS 9.
They also exist on Yosemite.
And we will also be using AVAudioEngine in some
of our example code we will be showing today,
in particular the AVAudioUnit class
and AVAudioUnitEffect class.
Those have been available since last year's OS releases.
So about compatibility now.
This is how things look now in OS X.
We have our existing version 2 Audio Unit hosts
and existing version 2 Audio Unit implementations.
The hosts start their communication
with audio component instance new,
and our implementations are built
on audio component factory functions.
We have a new set of APIs here,
so we will have new hosts using those new APIs.
And new Audio Units implemented using those new APIs.
Hosts will communicate with the class AU Audio Unit.
New version 3 Audio Units will subclass AU Audio Unit.
So that's two separate APIs.
What are we going to do to be compatible?
We have built bridges between these two APIs.
So thanks to these bridges, we will find
that new version 3 hosts should be almost completely compatible
with existing version 2 Audio Units.
And conversely, existing version 2 hosts would only need minor
source changes to work with new version 3 Audio Units.
And I will detail those API changes in a little bit.
So now I would like to give you a demo
of a new Audio Unit working
in an only slightly modified version of Logic Pro.
I have a little session here, it has a drum loop built in.
And here I'm going to apply an Audio Unit to this track.
So here are all of the Apple built-in Audio Units.
And here I have a new demo Audio Unit called v3 Distortion.
So I can open this Audio Unit.
I can find the preset I like, and we can hear Logic playing
through this Audio Unit.
There it's dry.
Now, if I go to activity monitor here,
we can see that this distortion Audio Unit is running
in a separate process, AU v3 distortion.
It's consuming a little bit of CPU.
It has some threads running.
Now, suppose this Audio Unit has a bug in it, and it crashes.
Well, I can simulate that here in activity monitor.
I can force quit it.
And notice in Logic, the view went blank
but the music kept playing.
So here is a diagram of what we were just looking at.
That's a slightly modified version of Logic Pro,
but it's still basically communicating using the existing
version 2 API, which is bridged to AU Audio Unit and in turn,
in that separate extension service process,
we saw the distortion units AU Audio Unit subclass running
along with its custom View Controller.
In the Logic process, there is also a View Controller,
and you see how these are bridged
across the process boundary.
Now, I'd like to talk about hosting Audio Units
and I will show you an example that uses the version 3 APIs.
We have sample code called Audio Unit v3 Example.
I checked a couple of hours ago, but it hadn't appeared yet.
I hope it comes out today.
In this sample code project, you will see there are a number
of targets and one of them is called AU Host.
Now, this application is fairly simple and straightforward,
but it shows how to find and open Audio Units that are
on the system, how to connect them together
into a rendering chain, how to select Audio Unit presets,
and how to open an Audio Unit's custom view.
So in the AU host app, we have something called simple play
engine, which is a Swift class that uses AVAudioEngine.
It uses an AV audio player node connected
to an AV Audio Unit effect.
That AV Audio Unit effect
in turn exposed an underlying AU Audio Unit,
which is the maiden class of the version 3 Audio Unit API.
We have the player to the effect,
to the mixer, to the output.
That's how the simple play engine makes sound.
We will also see how to use the AV Audio Unit component manager
class to select from the AV Audio Unit components
on the system and use that to control what kind
of AV Audio Unit effect gets selected.
So let's get into a little bit of code,
but first there is a very fundamental data structure here
when working with Audio Units.
We have the audio component description
and its first three fields: the component type, type,
subtype, and manufacturer.
That tuple uniquely identifies an Audio Unit in the system.
The flags are also important.
They are partially populated by the audio component,
and there are new ones populated by the system.
We will describe some of those as we go along.
The important thing here is this is the key
that identifies the plug-in.
So to find Audio Unit components on the system,
the first thing we do is create an audio component description
that contains a wildcard.
Here we say the component type is effect.
That's not a wildcard, but we have component subtype
and manufacturer of zero.
Those are wildcards, so we have built a component description
here that identifies any effect.
And then we can take that any effect component description
and pass it to AV Audio Unit component manager,
and it will give us back all of the effects
on the system matching that wildcard.
So here we get an array of AV Audio Unit component objects,
and those contain things like the name, tags,
also the audio component description of that unit.
So here we have got an array of components.
We can pass that back to the UI,
and in turn the UI can call this method in the simple play engine
to select one of these previously vended
So here it gives us a component.
We are going to fetch the audio component description
out of that, pass it to an internal method, and the guts
of that internal method is here.
We are going to call a new class method of AV Audio Unit.
And this method asks it to create an instance based
on the component description we have here.
Now, this is an asynchronous function, meaning it's going
to go off and start instantiating it,
and then it's going to call this closure
when it actually has instantiated the Audio Unit
and we are ready to use it.
So here we are in our callback.
This is Swift closure syntax.
We have our AV Audio Unit.
And then we can attach it to our engine.
We have stored it into a member variable, the effect.
And now we have an AV Audio Unit know that's the effect.
We are going to patch that into the engine.
We will disconnect the effect from the main mixer,
then connect from the player to the effect.
And then from the effect to the main mixer node.
So now we have got an effect in our play engine.
And now we can store the actual AU Audio Unit.
That's the plug-in, and here we can do all kinds
of interesting things
like manipulate the component's effect, presets, and parameters.
For instance, here, we will just get the list of factory presets.
And this too can populate a field
in the table view -- rather, in the UI.
So the user can choose the factory preset.
And finally, I would like to show you how
in the app I will show you in a minute,
we can get the Audio Unit's custom view and embed it
into the host application's view.
Here we are in the View Controller of the host,
so we are going to ask the play engine, give me your Audio Unit,
and then we are going to ask the Audio Unit
for a View Controller.
When it's done with that, it will call us back
with a View Controller that we can embed in the host's view.
Okay. I would like to bring up my colleague Michael Hopkins now
to show you this app actually running now.
MICHAEL HOPKINS: Thank you very much, Doug.
I'm delighted to have this opportunity today
to show you this AVAudioEngine-based v3 host
application running on an iPad here.
As you can see, I'm going to launch
that by tapping the icon for the host.
And on the left-hand side of the screen we have a list of all
of the effects Audio Units that are present on the system.
And this includes both the built-in Apple audio
component-based effects as well
as several new extension-based v3 Audio Units I
At the top of the screen, I have a Play button that I can tap
to toggle the playback of a drum loop.
Now, let's see how I can apply some effect nodes
and add them to the graph.
First, I will play this with no effect
and then I will add a couple effects
so you can hear that working.
With the high pass filter, it's filtering out almost all
of the sounds of the cymbals and other higher frequencies.
A delay, which is a little bit hard to hear in this room.
And I'm going to go ahead and stop that.
So now I would like to show you
for the first time an extension-based Audio Unit
running on this iPad.
That is the distortion demo.
When I select that, now you can see the list of all
of the factory presets that the Audio Unit is publishing.
These include some drum-specific ones as well
as some really crazy, wild effects like alien chatter.
Now, as Doug mentioned ,
v3 Audio Unit can have a custom view on iOS.
And I'm going to show you that.
I'm going to go ahead and tap the View button.
And what we have done is we have loaded that View Controller
from the Audio Unit and I have installed it
as a child View Controller within our application context.
So for the first time, we have a built in Audio Unit
with a UI running in our host.
We have a large slider, excuse me, a large knob that I can use
to control the amount of distortion.
And let me go ahead and play that for you
so you can hear that in action.
It's really a lot of fun.
It's an amazing experience to be able to have
that Multi-Touch UI working fluidly in a host application
without having to go through all of the hassle of switching
out to another application, doing some tweaks,
switching back to your host,
starting recording, switching back.
Now, you won't have to do that ever again.
And I'd also like to point out further
that this is the same Audio Unit that you saw running in Logic
in Doug's earlier demo.
In fact, the source code for the Audio Unit is identical.
No changes were required.
The drawing code is also very similar because I chose
to write this using Core Animation
so that API is almost fully portable.
The only changes that were necessary
to bring this Audio Unit to iOS are in the event model,
whereas we have had to use the touch events in UIKit
versus the AppKit mouse events on the desktop.
So really you guys have an opportunity
to with only a few changes to publish an Audio Unit both
on the desktop and on iOS.
Thank you very much.
Back to you, Doug.
DOUG WYATT: Thank you, Michael.
So I would like to talk about using Audio Units
in your host applications in situations
where you are not using AVAudioEngine.
We have a similar method on the AU Audio Unit class
to asynchronously create an instance
of the component description.
You see that there.
We also, for those of you with existing version 2 hosts,
a minimal translation path is
to start using audio component instantiate.
We will talk about that in detail in a bit.
Now, I would like to talk about the subject
of extension service processes
versus plug-ins loaded into host processes.
Now, as anybody who has worked with Audio Units is aware,
of course, with our existing plug-in model,
the plug-ins are always loaded into the host's progress,
and this remains true for version 3 hosts.
If it's a version 2 existing plug-in, and that might be one
of the Apple built-in ones on iOS
or it could be a third-party one on OS X, but in any case
if it's a version 2 Audio Unit, regardless of any other factor,
it's always in the host's process.
Now, version 3 Audio Units have a slightly more complicated
By default, version 3 Audio Units are loaded
into a separate extension service process.
And this is the diagram we saw before with Logic.
This is true, again, whether it's a version 2 host
or a version 3 host.
Now, on OS X only it is possible for the plug-in
to be loaded directly into the host's process.
Now, for this to happen, both parties have to opt in.
The host when instantiating the Audio Unit has
to pass this option to any
of the asynchronous creation methods we just saw,
and you see the name of that new flag there called Load
And the Audio Unit also has to be packaged specially
and consent to this with a plist entry called Audio
So if both parties do opt in,
then the framework will actually load the plug-in
into the host's process.
So the host will be communicating directly
with the plug-in's AU Audio Unit subclass.
Now, as a host author, why would you want to do this?
There is a tradeoff here between safety
and performance is the reason.
Of course, it's a security risk to be loading third-party code
into your app, and if it crashes inside your app,
then users might blame you instead
of the misbehaving plug-in.
But on the other hand, we have performance reasons
where you may want to load plug-ins into your process
if you are a host, because there is some overhead
to communicating with that separate extension
And we have measured that as being on the order
of 40 seconds microseconds per render cycle ,
o you can do the math to figure out how significant that is
in the context of your host.
You have some number
of out-of-process plug-ins you might be communicating with,
so you have to add that up.
And there is also the factor
of how much audio you are asking them to render.
For example, if you are rendering at a very low latency
of 32 frames, that's a 1 millisecond render interval,
so overhead of 40 microseconds could be significant
at 5.5 percent.
So that's your tradeoff if you are a host author.
I mentioned earlier
that existing version 2 Audio Unit hosts need a few changes
to work with version 3 Audio Units,
and here is what has to change.
I mentioned the audio component description flags,
and in there, the component flags.
There is a new flag called Requires Async Instantiation.
That's set for most if not all new version 3 Audio Units,
so if you see that flag set in the component description,
you have to use the new Audio Component Instantiate method
instead of Audio Component Instance New.
Now, similarly in an existing v ersion 2 host,
if you want to access an Audio Unit's View Controller,
then you also need to use a new asynchronous method to do that.
There is a new property, Request View Controller.
It is also asynchronous.
You can read about the details of that
in Audio Unit properties.h.
So about these asynchronous methods.
You can use the new methods with version 2 units,
but you must use them with version 3 units
when the flags are set.
And the reasoning here, well, the big sort
of externally facing reason is that it helps responsiveness.
If it's going to take half a second to instantiate
that Audio Unit, well, if you unblock the main thread,
your host applications, meters,
or other animations will keep drawing smoothly.
Now, especially when updating existing code --
and this was the first thing I did
when testing internal test code -- it's tempting to sit there
and wait on the main thread
for the asynchronous operation to complete.
Well, don't do that, because not only will you block any graphics
you are doing, but you will also block some underlying operations
in the framework that are actually required
for the Audio Unit to be instantiated.
So you will deadlock if you block on the main thread.
Don't do that.
Now, I would like to switch gears from talking
about hosting Audio Units to creating Audio Units
with the new version 3 API.
First, a few words about app extensions
since the new Audio Unit model is based on app extensions.
App extensions are bundles with a file type extension of .appex.
Xcode will build them into an app's plug-ins directory,
and we saw how they get loaded by the system
into separate extension service processes.
You can read all about the nuts and bolts of app extensions
in the app extension programming guide.
Now, our new sample code project, Audio Unit v3 Example,
contains a sample Audio Unit implementation called
Filter Demo, when you look
at that sample project, has three targets.
It has what we call the containing app,
and what it contains is the app extension as well as a framework
where a lot of its common code exists.
Both the app and the extension link against this framework.
Now, inside this framework, we have two main classes.
There is AU v3 Filter Demo,
that's the AU Audio Unit subclass,
and the Filter Demo View Controller
that controls the custom view of the Audio Unit.
Now, what's cool about doing things this way is
that while we are developing our signal processing and view code,
we can do this all in the context of our own app
so we are debugging not in a separate SPC service process,
but we are debugging
and developing our code right there interactively
in our own app.
We also let our app look like something
when the user opens it.
It's not just a plug-in for somebody else.
And we are not duplicating any code
to be able to accomplish that.
There is one extra bonus here that on OS X, if we want to,
we can designate this framework as being the bundle
that a host process can load into itself.
So let's look at the app extension.
It has an info plist with important entries.
The NSExtensionPointIdentifier tells the system what kind
of extension it is, the main storyboard tells the system,
when you launch my extension service process,
open the storyboard.
And finally, there is an Audio Components array
that tells the system, here are the audio component descriptions
that I am registering.
Just a quick reminder here, in your storyboard, you will want
to be sure to specify your custom class.
You may need to specify the module if you are building it
into a separate framework like we are here.
Then the extension actually has no code in it other
than this little bit of dummy code
to keep it from being empty.
We have to link against the Filter Demo framework.
All of the good stuff is in there,
and here we just have a global variable referring to it.
Let's move onto the framework now.
So the main class
in the framework is the Filter Demo View Controller.
In extension terminology,
this is the extension's principal class.
Whenever the extension is created or loaded,
the system will create an instance
of that principal class.
And it's got two main jobs.
It in turn creates the AU Audio Unit subclass and,
as a View Controller as you would expect, it creates
and manages the plug-ins custom view.
Here is the class declaration
for the Filter Demo View Controller.
It derives from AU View Controller,
which is either an NS or UI View Controller basically,
and it also implements a protocol called AU Audio
That's a simple protocol and implements exactly one method,
Create Audio Unit with Component Description.
And the job of this method is
to create the AU Audio Unit subclass.
And here it is, the AU v3 Filter Demo.
Now, let's look at that AU Audio Unit subclass.
So for reasons we will get into in a bit,
these are actually embedded C++ classes or objects.
The filter DSP kernel is where all of the math happens.
We will listen to it later.
It's a little more interesting than looking at its code.
We have some code here dealing with the buses.
This is an effect.
It has one input and one output, and our base class is going
to want us to provide arrays of buses
so we have numbers to support that.
And we have something called a parameter tree.
We will see what that is in just a second.
Here is the initializer.
So the first thing we do is initialize our input
and output buses, and then we wrap them in bus arrays.
And these arrays each contain a single bus.
And now we are looking at parameters.
So every parameter is an object, and you can think of this object
as kind of the bridge between your implementation
and the host.
In the middle, there is the parameter object.
This is a simple low-pass filter,
so it's only got two parameters,
a cutoff frequency and residence.
Every parameter has an identifier,
so here we are saying it's cutoff.
It has a localizalbe name.
We are being bad and not localizing it here.
It has an address.
We will talk about that in a bit.
Arrange, and some units, and flags which you will recognize
as being almost identical to what we do
with version 2 Audio Units.
So here we have created our first parameter.
We will create our second one almost identically.
And then finally we can create our parameter tree,
passing an array of those two parameters.
Now, we have our parameter tree, and we want to wire it up so
that it's connected to our DSP code.
And the way we do this is install a block
into the parameter tree called the Implementer Value Observer.
So this block will get called any time somebody,
whether it's the host or our own view, changes a parameter,
And so in response to that change, we will simply set
that new value on our filter DSP kernel
so that it takes immediate audible effect.
Now, in the other direction, there are times
when the tree needs to refresh its value based on what we have
in our signal processing.
That's what this block does.
It fetches the current value from the DSP
and returns it to the tree.
Next, this is an important override method.
If you're familiar with the version 2 Audio Unit API,
this was called Audio Unit Initialize,
which is not a good name choice in the Objective-C world.
So we decided to make it very specific.
What was initialize time is really prepare to render
and allocate the resources that are associated with rendering.
So there are things like buffers, DSP state, and so on.
So the first thing we do here is called the base class method.
Then we can ask our input bus to allocate some memory
for audio input to the plug-in,
and we can initialize our signal processing code here based
on the current channel count and sample rate of the output bus.
So entirely parallel, we have a method that's called
to deallocate render resources.
And here, too, we call the base class
and basically undo whatever we did when allocating.
So the process of rendering works through a block
that gets called every render cycle, but we are asked
to provide this block before starting to render.
Here we are going to capture our C++ members
into local variables that are pointers.
Now, the reason for this is that we are going
to be running our block for rendering in a real-time context
where it's not safe to access any Objective-C objects
because the runtime could block and cause an audio glitch.
So, again, we are just going
to capture our C++ member variables.
And then we can return the block.
It returns AU Audio Unit status.
And if you are familiar with the version 2 API,
the parameters are largely the same.
There is a time stamp, a number of sample frames,
an output audio buffer list,
and here is something new called the real-time event list head.
I will talk about that in detail, but it relates
to scheduled parameters and MIDI events.
And finally, the Pull Input block.
This is how the host tells us, the implementer
of the Audio Unit, where to get input from.
So in the guts of the input block,
the first thing we will do is pass that Pull Input block
to our input C++ object and ask that input object
to fetch the audio input for this render cycle.
Then we are going to do some housekeeping with buffers.
We will send them down to the DSP state, and finally,
we ask the DSP state to process the audio for this render cycle.
It's already been informed of the buffers,
and we are just going to give it a time stamp, the frame count,
and the linked list of real-time events.
So that's the guts of this Audio Unit,
there is not a whole lot of code.
There is a lot more code
that does the actual signal processing,
but as I mentioned before, it's better to listen
to that than to look at it.
So I would like to bring Michael Hopkins back
up to show us the AU v3 Filter Demo.
MICHAEL HOPKINS: Thank you, Doug.
I'm going to go ahead and start with the app container
that contains the extension.
You will see first on the screen
in the left-hand side is our Filter Demo,
which we have distributed as sample code.
To the right of it is the distortion demo application
that I showed you earlier.
I will go ahead and launch the Filter Demo.
Now, at the top of the screen,
you will notice the two parameters
that Doug talked about.
We have the cutoff in the residence parameter,
and these are represented in our UI
with a slider and a text field.
And this portion of the UI is actually contained
in the application, whereas the larger area in the main screen
with the graph is actually our embedded view
from the Audio Unit.
I can go ahead and change the value of the parameters
by dragging the slider.
And what's happening here is the application is changing the
value of that parameter.
And the view is listening for changes to that parameter,
and then it's updating.
As you will see, that update is live.
Conversely, I can interact directly
with our embedded Audio Unit view by tapping
in the graph and dragging.
And you will notice that as I drag that with my finger,
the application is receiving notifications
that the parameters have changed and it's updating in turn.
But that's kind of a boring demo without any audio going
through it, wouldn't you say?
Let's go ahead and take a listen to that.
I could do this all day.
You got time?
Now, it's really, really cool to just, the fluidity.
Just how fun it is to use your fingers to just be able to play
with that in a Multi-Touch UI.
Another thing that's cool about this is
because we have designed the user interface in such a way
that it can adapt to any size that it's embedded in,
we can actually take this iPad and we can rotate it sideways,
and you can see that now the user interface is updated.
And from a portrait view to a landscape view, and vice versa.
Now, we are doing that because we are supporting Auto Layout
and we are looking at size classes, so that's really great.
But what happens when we go and we take this and we put it
into our host app, which has a much smaller amount
of screen real estate dedicated to plug-ins?
So I will go ahead and switch back, and we are going
to open the host there.
I'm going to get rid of the beautiful distortion demo
and embed our Filter Demo view.
Tap on View to load that.
And now you can see that that's being loaded
in a constrained vertical space and a very wide horizontal space
yet it still works as you would expect,
and none of the labels overlap.
It still works exactly as we would expect.
So this is a fantastic new technology,
and we are so excited to be able to finally bring it to you guys
and I can't wait to see what do in your own iOS apps.
DOUG WYATT: Thank you, Michael.
Just a few words now about the containing app in general.
It is a vehicle for your plug-in and helps you
with rapid iteration and development, but you can think
about putting some extra things in that containing app.
We saw with the Filter Demo that it has the simple play engine,
you may for whatever reason want a more complex play engine
of some sort.
This is also a good place, the containing app,
to try putting creative touch controller.
There may not be room in a plug-in view
for your full touch controller.
Maybe there is, but you might think about having extra,
an extra-large version or something
in your containing app.
The app is also a good place for any documentation or help
that would make the plug-in view a little dense.
A few final words about creating an app extension here.
So if you are going to build a framework to be loaded
in process on OS X, despite what we are doing here with Swift,
we can't recommend that you do this on OS X
because the Swift API is subject to change.
If you build your plug-in against one version
of the Swift runtime and you are loaded into a host that happens
to be using another version, there could be collisions,
and that would be bad.
We realize when you look at the sample code here and you try
to build your own plug-ins, there is a lot, you know,
there is three related targets that have to be built properly.
It's a little bit complicated.
We do plan in Xcode's template,
but for now you can copy liberally from the Filter Demo.
Okay. Now I would like to talk in general
about the modernized AU Audio Unit API from both the host
and the implementation sides.
I would like to compare the way that properties are handled
in version 2 versus version 3.
In version 2 Audio Unit API, we have scope-
and element-based properties.
A large number of properties are in the global scope
so you have a bunch of code where you type.
K Audio Unit scope global, element 0.
And it's painful, especially from Swift,
where we have property values that are void pointers.
You end up typing unsafe mutable pointer all over the place,
and that makes my head hurt.
We have these functions with long argument lists.
And by comparison in the version 3 API, well,
properties are properties.
We use a dot syntax in Objective-C and Swift,
so you can write AU.maximum frames to render.
We also implement our classes to be key-value coding
and key-value observing compliant so you can use value
for key, and add observer for key path.
We also added a special KVO method to the bus array,
add observer to all buses, so you don't have
to simultaneously be watching for buses to come and go just
so that you can add KVO observers on them.
That can be a painful cycle to chase.
Speaking of buses, they are full-fledged objects
in the new API.
We have the AU Audio Unit bus array.
The AU Audio Unit has an array of input buses,
an array of output buses.
And the buses have two major properties.
They have a format and a name.
The format is manipulated by the host.
We are able to reject formats that we don't like.
We use the same formats
in version 3 Audio Units as version 2.
Let's look at parameters.
We have some of the same problems here
in the version 2 API with parameters
as we did with properties.
We have these unwieldy scope element ID tuples.
And furthermore, in some very complex AUs we just didn't have
We have these functions with long argument lists again,
and we also have a complicated AU event listener API.
In the version 3 API, I hinted at this earlier
with the parameter tree and the Filter Demo.
Well, it is a full tree.
Parameters can be grouped, and here we have an example
of a simple analog synthesizer emulation.
It has groups for oscillator, filter, and amplifier.
The filter and amplifier have envelope groups beneath them.
And the most brightly colored boxes beneath are
So the waveform octave, filter cutoff and resonance,
and the envelope attack, sustain, and release.
So these boxes are all nodes, whether they are groups
or parameters, and every node
in the parameter tree has a unique and permanent ID.
This is like a C identifier.
So using these unique IDs, we can use KVC to go
and find a parameter we are looking for,
such as oscillator.wave or filter.envelope.attack.
And this would be a lot more flexible
for these very complex audio units
that have very large parameter trees.
Now, you will notice
that parameters have numeric addresses
and that they are 64 bits, but we do have to treat them
as transient in any situation where we aren't the one who made
up that addressing scheme.
So that means if I'm a host application and I want
to record some parameter automation, I should record
that automation using the key value path --
I'm sorry, the key path of that parameter and not its address.
I alluded to this earlier.
The AU parameter object is the focus of communication
for parameter values between hosts and views,
and on the other side, the Audio Unit implementations.
Now, from the host's point of view,
the parameter object has properties like its value,
also a minimum and maximum value, and so on.
So we can set and get parameter values using dot notation
as you would expect.
Now, we can also set values in such a way
to prevent a feedback loop, which is useful both
for performance and for keeping the UI smooth.
We don't want to be getting notifications
that are slightly different
from what we are doing as we move a slider.
So the set value method accomplishes that.
And that token is obtained from a method to add an observer
to a parameter, or its tree, or a group in the tree.
And when we do that, then we can get called back
from the parameter.
That's what we see at the bottom there.
There is the block called the AU parameter observer,
and it passes us the address and the new value
of the parameter that changed.
As for the implementation, we saw this in the Filter Demo.
It has the implementer value observer
and value provider blocks.
Now, in Filter Demo,
it installed these blocks on the tree.
It's also possible to install them at any level of the tree,
even on individual parameters.
I would also like to show off what we have done
with parameter scheduling
because I think this is a big improvement
over the version 2 API in this area.
We have the host and the implementation dealing
with things somewhat separately,
but here we have the AU Audio Unit base class doing some help,
it's helping us implement this.
So the host can obtain from the AU Audio Unit a block called the
schedule parameter block.
And at render time, it can call this block to change parameters
in sample-accurate way.
So the first argument to do schedule is a sample time,
the parameter value can ramp over time
if the Audio Unit has advertised it as being rampable.
For example, the Apple Mixer does this.
And the last two parameters, of course,
are function parameters -- are the address of the parameter
to be changed and the new parameter value.
Now, things are a little bit different
on the implementation side.
We don't just get a pass-through call from the host.
Instead, the base class is going
to fetch the internal render block, which we saw
in the Filter Demo, and it's going to pass
that render block the real-time events
that pertain only to that render cycle.
So the base class is maintaining the full schedule of all
of the pending scheduled parameter changes
and just parceling out the relevant pieces of it
to the Audio Unit at rendering time.
So that's parameter scheduling,
and we have done the exact same thing with MIDI events.
The host fetches a block
from the Audio Unit before starting to render.
It calls that block at render time.
Now, you will notice here we have added a function argument
called cable where,
in the version 2 Audio Unit API there is only one MIDI cable
with 16 channels, now we have 256 virtual MIDI cables.
So if you have an Audio Unit
that wants huge sample banks, you can do that.
You can address them all on virtual MIDI cables.
On the implementation side for MIDI events,
this is exactly the same as for scheduled parameters.
The base class AU Audio Unit is maintaining internal schedule
and passing events to the internal render block
through the real-time event list only during the render cycle
during which they are supposed to take effect.
So we think this is a big improvement.
It saves the implementer a lot of work.
Now, about rendering in general, we are still using a pull model,
meaning that an output unit pulls a mixer, pulls an effect,
pulls another effect, pulls the player.
Audio flows back down through the chain.
One difference here is in the version 2 API,
the Audio Unit needs to maintain some state.
It needs to have a notion of whether it's getting its input
from another Audio Unit upstream or a function callback.
Now, in the version 3 API, it's much simpler,
the AU doesn't have to maintain that state.
This callback, as we saw in the Filter Demo,
comes from the host, and it is passed during every
But otherwise, the APIs are pretty much functionally
identical, and this lets us bridge very efficiently
Now, if your host is calling AU Audio Unit directly to render
as opposed to using AU graph or AVAudioEngine,
here you will want to call allocate render resources
as usual and then hold onto the render block.
You can call the render block to render.
It looks very similar to the internal render block.
It's worth reviewing here some rules
about the audio buffer lists that appear at render time.
Now, the host provides an output audio buffer list,
and in that audio buffer list the M data pointer can be null.
The Audio Unit must replace this with an internally owned buffer
and at the same time, the AU has to promise
that that buffer will remain valid
until the next render cycle.
This is all, by the way, exactly the same
as with version 2 Audio Units, I'm just reemphasizing it
because it's important.
Now, in the render block, we have some rules, similar rules
but not the same, about input buffers.
The host provides that input block, the AU calls it
for input, and when the AU calls that block for input,
it has to supply valid audio buffer lists
with non-null M data pointers to that block.
Now, the host is allowed to replace those pointers to memory
that it owns and can promise to keep valid
until the next render cycle or deallocate render resources,
and all of this accomplishes an important goal,
which is to absolutely minimize copying.
Okay. Here is the scary slide for those of you
who are writing code to run in rendering context.
So audio rendering almost always happens
in a real-time thread context.
And this is a restrictive environment
because we can't allocate memory,
which means that we really shouldn't even be calling
dispatch async, for instance.
And in fact we can't make any call at all which might block,
for example, taking a mutex or waiting on a semaphore.
The reason is if we do block and we block for any length of time,
then the audio rendering thread
in the system will miss its deadline,
and the user will experience that as a glitch.
So we have to be very careful when both using and calling or,
I'm sorry, both using
and implementing these render blocks.
So you will see in the Filter Demo how we went to some lengths
to not capture our Self object
or any other Objective-C object for that matter.
In that block, we avoid the Objective-C runtime
because it's inherently unsafe.
It can take blocks.
Unfortunately, the Swift run-time is exactly the
So this is why in the Filter Demo you'll see we have C++
objects and we capture pointers to those C++ objects.
Now, if you are allergic to C++, you are free
to do the same thing in plain vanilla C,
although I'm not sure why you would want to do that.
Enough scary stuff.
I would like to bring up Alec Little now
to show Audio Unit extensions in Apple Music creation apps.
ALEC LITTLE: Thanks, Doug.
I'm Alec. I work on the music creation applications for Apple,
things like GarageBand and Logic.
We are excited about the new Audio Unit extensions.
We think they will add real power and creative possibilities
for developers and users.
So I wanted to talk a little bit
about what some of our plans are.
So first of all, we plan
to support the Audio Unit extension, of course,
in all of our main applications.
So that's GarageBand iOS, GarageBand Mac, Logic Pro X,
Logic Pro 10, and Mainstage.
So what I thought we would do today is look at some examples,
some pretty pictures, if you will, from GarageBand iOS.
And this is just preliminary stuff,
but I think it will give you an idea of kind
of what we are planning to do as a host
to support Audio Unit extensions.
So first of all, we are going to be supporting AU instruments.
So the example I'm going to be giving is about how we're going
to implement those AU instruments.
So first of all, just a little graphic to explain
about what we are going to do, something simple here,
but GarageBand is going to request
from the View Controller a custom UI dimension
that we will talk about in a second.
Pass MIDI events over to the Audio Unit and then
of course receive audio back over the audio bus.
On to the promised pictures.
So GarageBand, our main launch screen launches
into what we call our touch instrument carousel.
We have all of our touch instruments here: keyboards,
drums, smart guitars, all of those things.
Then if you see on the left there is this container,
and if GarageBand is seeing
that there are Audio Unit instruments installed
on the device, we will show that container.
I can swipe over to that container,
and that's where my Audio Units live.
If I tap on it, then we will see all
of the Audio Unit instruments installed on the device.
Now, if I tap on one of those instruments,
we will show a big gray box and a keyboard -- no.
We will show your custom UI up there
in that nice embedded view inside GarageBand,
and I think that's the coolest part of this whole thing is
that we get to show the actual identity
of your Audio Unit inside our host there.
And we will be providing our standard GarageBand keyboard
for you to be able to play that.
We will be recording the MIDI and receiving the audio back.
So that just kind of brings up a pretty obvious point.
When you are providing these custom UIs,
make sure you are not putting any sort of custom, you know,
MIDI controller-type device there
because we won't be capturing your MIDI in GarageBand.
Just a quick look what we plan to do on the phone.
Is, again, we have a lot more limited screen real estate
there, so there is a button in the top-right corner,
which pulls up the controls view,
and there is the custom UI.
So all of the control there, and a little bit of room
for the user to still play on the keyboard down at the bottom.
So here is the most important slide probably from me today,
and this is the dimensions we will be requesting
from the View Controller.
So you want to have your stuff look good in GarageBand,
pay attention to those dimensions,
and we will do something pretty cool together.
So, again, we think it's going to be really exciting to be able
to see and have the users be able to play
with the actual interface
of your Audio Units right inside GarageBand,
and we are really excited to work with you guys to come
up with really cool stuff!
DOUG WYATT: Thanks, Alec.
So I imagine you may have questions at this point.
I would like to try to anticipate a few of them.
What about inter-app audio on iOS?
This is a couple of years old now.
And there is a number of apps that have supported it.
Well, from our point of view, this uses a small subset
of the version 2 API, and it doesn't support a bunch of thing
that people have asked us for,
like parameter support, presets, and so on.
And we get these requests and I think, well,
we should have a full plug-in model,
which is what we have now.
So while we are not deprecating inter-app audio,
we do see the way forward to add all of these missing features is
through Audio Unit extensions.
Now on OS X, you may be wondering
about the compatibility story
if you have existing hosts and Audio Units.
The bridges should save you from a lot of pain.
They are compatible, and we have gone to a lot of work
to make things work as well as we can.
So I would recommend that you update to version 3 when you can
or need to for some feature.
Maybe you want to redo the way you handle MIDI events
or scheduled parameters, for example.
We do have a shortcut for porting.
It's called AU Audio Unit v2 Bridge.
This is an AU Audio Unit subclass that is implemented
on top of a version 2 AU, so you might be able to start with that
and evolve to a more fully native implementation.
And as Michael mentioned earlier,
version 3 Audio Units are largely cross-platform
between iOS and OS X.
The signal processing code
in AU Audio Unit should be absolutely fully portable
because there are no UI implementations,
or dependencies, rather.
AU View Controller derives from either UI or NSViewController
so you get a little bit of insulation, but you will
at some point run into platform-specific UI.
We are running out of time, and there is way more
than I could possibly go through in an hour here.
At this point I would like to refer you to your header files
in the Audio Unit framework.
You do need to link AudioToolbox for historical reasons.
The main header file is AU Audio Unit.h but there are others,
AU View Controllers in the Core Audio kit framework,
and as we mentioned, there is the AVFoundation framework
with AU Audio Unit component.h.We have good HeaderDoc in all
of these, so I would like to urge you to check them out.
Finally, if you would like to use our Audio Unit's logo --
there is a white version, too --
you can go check out this link for a license.
And that brings us to the end.
So we have seen how we now have
for the first time a full plug-in model for audio on iOS,
and it's the same on OS X.
And, again, it lets you sell Audio Units in both the iOS
and OS X App Stores by packaging your Audio Units
as app extensions.
We looked at the simple host applications that are possible
with AVAudioEngine, and that's illustrated in our new sample,
Audio Unit v3 Example.
So I would like to encourage you to write bugs as you work
through the sample code, read the documentation,
and work on the great AU hosts
and implementations that I know you will.
So for more information, this was our session yesterday,
so thank you very much!
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.