Learn how CarPlay integrates with your car's infotainment system. Understand how CarPlay is designed to work with your car's resources including the display, speakers, microphone, user inputs, steering wheel controls, instrument cluster and sensors.
[ Music ]
Hello. My name is Tanya, and I want to welcome you
to the second session about developing CarPlay systems.
The video to Part 1 is also available.
So if you haven't watched it, definitely check it out.
In this session, we'll cover more detail
on how CarPlay is integrated
into a typical automotive infotainment system.
We'll start with a system overview, then talk about volume
and resource management,
and finish by discussing application state management.
Now, let's dive into an overview of the system.
If you watched the first video,
you will recall the software components you need
to implement a CarPlay receiver.
In summary, you need an IP-based link to the head unit either
over USB or over Wi-Fi, IP cline for data exchange,
an instance of the CarPlay Communication Plug-in
implementing the control protocol, audio framework
to play or record sound, and a video framework
to render the video stream.
However, those components are just one of the sub-systems
in a typical automotive head unit.
In addition to the CarPlay functionality,
your unit has its own native interface,
audio sources, and logic.
Both together build the complete head unit.
Then you add all hardware resources within the vehicle:
Microphone, speakers, displays, sensors, user input elements,
and you have the complete system.
OK, so we have the full system.
Now the question is how do those seemingly competing sub-systems
work with each other?
Let's understand it.
First, let's look at how sensor data is exchanged.
And we will take location data as an example.
Location information includes data from a GPS receiver
or sensor data like wheel speed or yield.
The data reaches the device through iAP2,
must be always available, and is requested
by the phone when needed.
Next, let's look
at the instrument cluster or heads-up display.
These secondary displays are used
to show metadata provided by the device.
Such metadata could include the currently-playing music track,
the active phone call, or starting with iOS 10,
the next turn-by-turn direction.
Again, the data is fired through the IP protocol
and exchanged on demand.
Now let's see what happens on the primary display.
Of course, to operate the content shown on the display,
the user must either use a touchscreen,
an app controller, or both.
So we can consider the availability
of the user input devices together with the display.
When the CarPlay UI is active,
the unit is rendering the video stream on the primary display.
And both the screen and the user input are used by the device.
But if the native user interface is shown on the screen,
then both the display and the user input are routed
to the native sub-system.
However, additional user controls on the steering wheel,
like the Voice Recognition button, Next/Previous Track,
may be linked to CarPlay even
when the native UI is shown on the display.
That way, Siri can be always launched
or the next audio track played.
OK, now that we looked at the user interface integration,
let's talk about audio.
Audio in the car can be roughly separated into three categories.
The first one provides access to the speakers and the microphone,
and it's used for phone calls or voice recognition.
Then we have audio playback
for any media content like radio or music.
And last, we have the alerts category which is used
for navigation prompts or any other higher-priority alerts.
Within the native sub-system,
these three categories are to be found again.
Within CarPlay, we have a Main audio channel
which includes phone call, voice recognition
and audio playback audio.
And an Alternate audio channel which is reserved
for higher-priority prompts and sounds
such as navigation prompts or new message notifications.
So the media channel can either play music coming from the phone
or media playback from one of the native audio sources.
OK, now that we have looked at media playback,
let's see what happens
when there is a phone call through CarPlay.
The phone call needs the audio channel which provides access
to both the microphone and the speakers.
So media playback is interrupted and the channels are switched.
And audio switched over to the channel, which provides access
to both the microphone and the speakers.
In addition to switching to audio channel,
the screen switches to CarPlay to show the phone call UI.
And metadata information
on the instrument cluster is updated as well.
Of course, if the native system also supports phone calls
outside of CarPlay, then that phone call is played and shown
on the in-car displays.
But while on a native phone call,
the user can be using Apple Maps for navigation.
So each time there is an approaching turn,
Apple Maps will play a sound
through the announcement channel to notify the user.
The phone call continues to be routed through the speakers
of the car, but the turn-by-turn notifications are also displayed
as they are mixed with the call audio.
Of course, in such situation, we would not want the user
to hear the phone navigation prompt, but just an alert sound
that an upcoming turn is coming.
And if the user wants to see where the next turn is,
the main display may show Apple Maps.
And last, after the phone call ends, music continues to play
through the native sub-system
as that was what the user was listening
to prior to the phone call.
Now, the navigation announcement will include the full-spoken
direction as the user is no longer on a phone call.
As you see, integrating CarPlay is rather a complex task,
especially in regards to sharing hardware resources.
But before we get into those details,
let's look at volume controls.
As you might know, a usual automotive head unit shows the
volume indicator while the user is operating the volume knob,
and that volume knob operates the volume
of the currently-playing audio.
The same principle applies for CarPlay audio,
but there is a different volume level setting for each
of the major CarPlay applications.
Let's look through them.
During Siri playback, the volume knob sets the volume specific
to voice interactions.
For an incoming phone call,
the volume of the ringtone is controlled.
When the user is on a phone call,
the phone call audio is controlled.
And if a next turn is approaching,
navigation prompts are played.
Then, the user can change the navigation volume.
And remember, media could be playing in the background
but its volume will be lowered during those prompts
and cannot be adjusted.
Once the announcement is over and music volume is ramped up,
the button controls the media volume level.
All right, now let's look at resource management.
We are going to take a look at which resources are managed,
talk about how they are managed,
and go over a couple of typical examples.
So we saw that there are multiple hardware resources,
but let's understand which one of them are managed.
And there's only two:
The mainScreen representing the main display in the car,
and mainAudio as the resource giving access
to the car's audio system.
Those resources can either be taken or borrowed.
When you take a resource, it belongs to you
for unlimited amount of time.
It's basically yours.
When you borrow a resource, you may use it for a while,
but you have to give it back after you're finished.
OK, so let's have the native user interface take
over the display permanently.
The user can be using native navigation, or listening
to FM radio, or adjusting some car settings.
What triggers such permanent switch to the native UI?
The native UI can take the screen
when the user presses a hard key or switches to the native UI
from the entry point within the CarPlay UI,
or by using the native voice recognizer to call
out a specific native application.
Now let's say the user selects the Apple CarPlay icon
from the main menu.
In this case, CarPlay takes over the display
as the user explicitly requested it.
Again, what can trigger such permanent transition?
Any hard key linked to CarPlay.
Or, as we saw, through any CarPlay button
within the native UI.
Or through Siri, for example, by saying, "Open Maps."
Now, there are other applications
that need to borrow a resource.
Remember, they just need it for a while and will give it back.
Which applications are we talking about?
That's phone calls, voice interactions,
notifications, or alerts.
Let's look at an example.
The native UI has taken the Main Screen resource
and is permanently showing on the display.
Now the user gets a call through CarPlay.
The CarPlay UI borrows the screen and is shown
for the duration of the call.
Once the call ends, we go back to the native UI and stay there
until further user action.
Then, let's look at the Main Audio resource.
Main Audio can be split in four major types.
Each type is used for a different CarPlay application
as it provides access to a different hardware resource.
Media is audio output only and is used for any media playback.
Alert is also only output
and plays ringtones and timer alerts.
speechRecognition is used for Siri
as it adds access to the microphone.
Same for the telephony type, which is used for phone calls.
And same for the default, which is used for undefined audio.
But don't forget the second audio channel used
for navigation announcement.
alternateAudio is not managed.
It is basically always available, so there is no need
to take it or borrow it.
alternateAudio is mixed with any of the audio types
within the mainAudio channel, and it is always accessible.
And with this, I'm going to hand off to Tom to talk
about the resource manager.
Hi. I'm Tom, and I also work on the CarPlay Engineering Team.
So now that you've learned which resources need to be managed
and you know you can either take or borrow those resources,
let's talk about how you go about managing them.
To be able to distribute the resources
between the two sub-systems, we need some type of arbitrator.
And we call such an arbitrator a resource manager.
So what does the resource manager do?
It has three main tasks.
First, it holds the current state of the whole system.
Second, it follows a set of strict rules to decide
which system will get the resource.
Third, based on the current state and the set of the rules,
it assigns the resource to one party or the other.
So let's see how this works in practice.
Let's say both the native UI and CarPlay need
to show something on the screen.
So both send a request to have ownership of the display.
Then, the resource manager checks the internal state
and decides which of the two should get access
to the display.
Let's say, in this case,
the native UI request had a higher priority
and the screen is assigned to it.
The resource manager sends a notification
that the screen can now be used by the native UI.
Only then the native UI can show content on the screen.
It's important to note that this state is not changed
until the resource manager has sent an update.
So where does the resource manager live?
Is it part of the native system or of CarPlay?
Well, when we designed CarPlay,
we asked ourselves the same question.
We considered the complexity
of exchanging detailed information about the resources.
We considered the future.
What, if any, of those rules needed to be adjusted?
What if new applications became available through CarPlay
that the vehicle had no existing designs to handle?
What solution would give us the greatest flexibility five years
down the road?
So we decided to implement a system where the complexity
on the native system is lower and it's easier to update
after the vehicle has been
in the customer's hands for a while.
Hence, the resource manager is implemented within iOS.
And because the resource manager is the component
with which the native system interacts,
we refer to the iPhone as the controller,
and the head unit as the accessory.
But don't forget, all
of the iPhone applications request the same resources
in the same way as the native UI would.
Next. What commands can you use to interact
with the resource manager?
You only need two commands.
The changeModes command is used to request
or release a resource, and the modesChanged command is used
to describe the current states.
changeModes is the notification sent by the head unit,
the accessory in this case,
to the resource manager, the controller.
This changeModes command states what the accessory wants
to do with the resource.
It declares why it needs it and who can take
or borrow the resource after it is transferred.
The modesChanged is the notification sent
by the controller back to the head unit.
modesChanged provides the current state
which describes who's the owner of the resources of the system.
It sends so the accessory knows
if a resource has been transferred
and the owner has changed.
Now we'll talk about how resource management works
in more detail.
We will start with the simple action
of switching audio from iOS audio to FM.
Then, we'll talk about native voice recognition,
then how to handle a backup camera case
where you would not want iPhone apps
to interrupt you unless we walk through an example
where Siri triggers music playback.
So let's look at playing FM radio.
Let's say iPhone music is playing
through the car speakers, then the user wants to play FM radio
through the native system and it's the Radio button.
The unit sends a changeModes request to have ownership
of the speakers, and it's taking it
as the user might continue listening to FM radio
for a prolonged period of time.
The controller assigns the audio to the head unit
and it sends a modesChanged notification.
And the head unit is the new owner of the audio resource,
so it can now start playing FM radio.
The mainAudio resource is permanently assigned
to the head unit at this point.
So, in summary, this example showed us
when to take a resource and that the native system should not use
that resource before it is the owner of that resource.
Let's look at the next example using native voice recognition.
FM radio is still playing from our last example.
Let's see what happens
if the user triggers the native voice recognizer.
The unit requests to temporarily own the display and audio,
so the transferType should be set
to borrow the audio and screen.
Both the mainScreen and the mainAudio are transferred
to the accessory.
And the native voice recognizer starts.
Once the voice dialogue is finished,
the head unit returns the borrowed resources
by sending an unborrow command.
And since the unit was playing FM radio before,
the resource is again assigned back to the head unit
and FM radio can resume.
Now you may be wondering, "Why would the accessory need
to borrow the resource if it was already the owner?"
Well, that's a great question.
Whenever the resource manager needs to evaluate a request
to change the ownership of a resource,
it must know what the current state of the system is.
The resource manager needs
to know why you're using the resources
so that they can take the right decision
if somebody else needs the resource later on.
Let's move on to our next example,
showing the backup camera.
Let's say CarPlay audio is playing
and the native UI is showing on the screen.
When the user switches into reverse gear,
the unit borrows the screen.
However, notice it also tells the resource manager
that the screen cannot be borrowed again.
The resource manager assigns the display to the native UI
and also notes the limitation.
So the backup camera is now shown.
CarPlay audio continues to play,
but now the user gets a phone call.
The iPhone cannot show any content on the screen
as the head unit has constrained the access,
but the ringtone plays through the speakers.
So how did that happen?
With the changeModes command,
the unit told the resource manager which rules to apply
by the resources in its possession.
By setting the borrowConstraint to anytime,
the unit allows the application to be able
to borrow the resource.
By setting it to user-initiated only events triggered
by the user can borrow.
With never, no application can use the resource,
as with this example.
As you see, the constraints have a major impact
on the system behavior, so it's very important
that they are used properly.
Only restrict access if there's a need
for immediate user attention.
Do not restrict permanent ownership
when CarPlay is connected.
So that was using backup camera.
And the last example we want to look at is what happens
when Siri plays music.
Again, we start with FM radio playing.
The user triggers Siri.
Siri is launched on the phone
and the resource manager tells the native system
that the resources have been assigned for Siri.
Siri shows up and the user can ask, "Siri, play some music."
Siri returns the display and audio resources,
but the music app now needs the audio resource to play music.
So the resource manager notifies the unit that mainAudio is now
with the phone, but the screen returns to the accessory.
The head unit does not continue to play FM radio,
and so iPhone audio is heard in the car.
So what did we see here?
On two occasions, the resource manager changed the state
without a request by the head unit.
And this is completely fine because the apps
on the device request resources as well.
And if they get them, the unit is notified about it.
So, how to handle this?
First, after each modesChanged, you check if you're the owner.
If you are, resume your activity.
Otherwise, you should wait for CarPlay to take action.
Second, do not ignore resource switches to the phone
when they are triggered by the phone.
OK, so far, we've looked
at examples showing how resources are being exchanged.
But there is another situation where both sides want
to accomplish the same task.
So let's look at how applications are managed.
We need to manage native and CarPlay applications
which have similar features.
But what are those applications?
They are active route guidance, phone calls,
and voice recognition.
We achieve those using appStates.
But what is an appState?
We use different appStates to record who is currently engaged
in the shared application that can only be active
on the accessory or the controller at any given time.
So the three different appStates are TurnByTurn,
PhoneCall, and Speech.
So let's have a look at how
that route guidance appState is managed.
When turn-by-turn navigation is started on the accessory,
it uses a changeModes command
to update the TurnByTurn appState to be active.
With this example, it is a last in win situation.
So the controller grants the TurnByTurn appState
to the accessory, returning a modesChanged command showing the
accessory as currently running route guidance.
Now let's look what happens
when the user sets a new destination using Apple Maps
while the native system is already guiding.
The user says, "Siri, take me to the closest coffee shop."
Now, the iPhone has started route guidance.
So the controller updates the TurnByTurn appState.
This is the instruction for the accessory
to stop its own native turn-by-turn guidance.
Now, the iPhone has started route guidance,
so the controller updates the TurnByTurn appState.
This is the instruction for the accessory
to stop its own native turn-by-turn guidance.
So now, the Apple Maps turn-by-turn directions are
shown on the display and announcement played over audio.
And there will be no conflict with the native guidance system.
So now when the Apple Maps turn-by-turn directions are
shown on the display and the announcement is played,
there will be no conflict with the native guidance system.
So that was route guidance.
Now let's look at a phone call.
If the unit supports a second phone
over Bluetooth Hands-Free Profile
and there was an ongoing phone call,
the call uses mainAudio exclusively.
The phone call cannot be interrupted.
So what happens if the user gets a phone call
on the CarPlay device?
The CarPlay call rings only on the device
as it should not interrupt the ongoing conversation.
This is managed using the PhoneCall appState.
And lastly, let's look at voice interactions.
If the unit has a native voice recognizer when it's running,
it has borrowed the resources
and is exclusively using the speaker microphone
and screen resources.
But it can be interrupted at any time by user-initiated events.
So if Siri is triggered by the user,
native voice recognition ends and Siri continues
on the in-car display.
The Speech appState is used to achieve this.
So, in summary, CarPlay relies on the same resources
as your native system and is designed to coexist
with your native user experience.
For a great CarPlay experience, consider resource handling
for each use case and follow the CarPlay design recommendations.
The CarPlay specifications are available
through the MFi Program.
If you haven't watched Developing CarPlay Systems,
Part 1 yet, I'd encourage you to go and check that out.
For more information on this talk,
please visit the URL on the screen.
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.