Core Motion enables your applications to access a wide array of sensors on iOS and watchOS devices. Discover how steps and activity detection can be used with user elevation changes to develop fitness apps. Explore ways to observe wrist rotation and effort and translate them to immersive health and fitness experiences.
A very good afternoon and welcome.
My name is Bharath Rao.
I'm an engineer with the Core Motion Team,
and today I'm really excited to be talking to you
about health and fitness.
In this talk, I'll be showing you how you can use Core Motion
framework to add some interesting and engaging health
and fitness features into your apps.
Before we get started, for those of you who are new
to Core Motion, I'd like to remind you to go check
out some of our past sessions.
They have some excellent information
about how sensors work, and how we use, in Core Motion,
those sensors to provide APIs
that help your apps observe stats and activity
and Device Motion, and a whole lot more.
Go check them out.
So with that, let's get started.
Here's what we have in store for you today.
I have an update to the Historical Accelerometer
that I would like to share with you.
We have a new pedometer events API,
both for the iPhone and the Apple Watch.
And finally, we are also bringing the Device Motion APIs
to Apple Watch, with watchOS 3.
First up is Historical Accelerometer.
Last year, we introduced the CM Sensor Recorder API.
With this API, now your apps have access
to raw accelerometer samples, or long durations of time.
And you can also get this access with very low power.
After your apps have requested the framework
to start recording these samples, they can get suspended.
And the OS will offer those samples,
even across device sleeps.
And when your app is ready to consume them, it can launch
and make a query, and get access to all
of those buffered samples.
In watchOS 3, we have a couple of important updates.
First of which is the greater duration.
Now, your apps have a 36-hour window in which they can launch
and get access to all of that Historical
The second update should not come as a surprise to you,
this is one of the most requested updates for this API.
So I'm happy to announce that in watchOS 3,
now the sample delay is as little as 3 seconds.
With these updates, now you can use Sensor Recorder,
not only to observe user activity, or at long durations,
but there are some real time applications
for which you can use the sensor recorder now maybe
to track sports activities or to record workouts at the gym
or even health diagnosis.
Imagine an app on the Apple Watch
that can detect hand tremors.
So now, whenever the user experiences some hand tremors,
they can launch the app.
Your app will be able to pull all
of the historical accelerometer samples, do some analysis
on them and get a report that says how severe that tremor was,
and what kind of tremor it was and share it with the user
and maybe even with the physician with consent,
so that it can speed up the diagnosis
and the treatment of such conditions.
So that was a great update
to Historical Accelerometer in watchOS 3.
Next up is pedometer.
CMPedometer is a versatile API.
It records stats and distance, and flights of stairs
that the users climb throughout the day.
So you could use it to build an all-day activity tracker
where you set some interesting goals for your users to achieve.
And that way you can motivate them
to lead a healthier lifestyle.
But where CMPedometer is really powerful is
in the context of workouts.
Take for example, the pace metric.
We have stride estimation algorithms that are running both
on the iPhone and Apple Watch and using those,
we are able to provide really accurate pace
and distance metrics, even
when the phone doesn't have a GPS signal.
Or when you go run with your Apple Watch,
and you leave the phone behind, even then,
the users get very accurate metrics.
The pedometer events API that we are adding today is actually
going to help you make those workout experiences even more
engaging and accurate.
Let's consider a typical running workout; an urban run.
As third packet [inaudible], one of the challenges
that you'll face with this scenario is how do you detect
all those starts and stops that the users experience
at intersections while they're running in a city?
So the user comes to a stop at the stoplight,
and your app will continue to accumulate all of that time
that they're just standing around.
So by the time they get to the end of their run,
now you have accumulated enough time, and you --
if you try to compute their average pace
over their entire run, you'll probably end up with something
that will resemble that of their granddad.
Or maybe granddad's gone faster than them, so never mind.
So what I mean to say is,
you'll end up with really inaccurate pace,
which is probably much lower than their running pace.
So one probable solution is you could provide a manual pause
and resume option.
But now, once they have paused, they will also have to remember
to resume the workout when they start running.
And if they don't, now all
of the running they do while their app is paused,
is going to be not recorded towards their workout.
So if they forget to pause, then they get inaccurate pace.
If they forget to resume, they lose out on distance.
So clearly, you need some auto pause and resume detection
that is not only accurate, but it also has to be responsive.
It has to feel like your app is doing a good job
of detecting those starts and stops.
At this point, you might be wondering,
why not just use GPS and steps.
After all, you have access to those in your apps.
If you have ever used GPS before, you know that you have
to do a considerable amount of filtering on it,
so that you can remove all the noise.
Which means that it is going to be again, slow to respond.
And with the steps that you receive from CMpedometer,
it has a built in delay.
And we do that because we want to avoid false positives.
We use steps to estimate stride.
And from that we compute distance and pace.
So it's very important for us
to have accurate step counts at the first step.
So in this release, we are giving you pedometer events
that is going to help you detect those starts and stops,
not only with good accuracy, but with low latency.
Our pedometer events solution, or the auto-pause
and resume solution uses a predictive algorithm.
This predictive algorithm was trained on user data
so that we can improve the likelihood estimate
of whether the user is moving,
or have they come to a complete stop?
Of course, all in a pedestrian context.
By doing this, now we are able to recover most of the delay
that is associated with the step counting algorithm.
But we are able to do so with pretty good accuracy.
I would also like to point out,
because the predictive algorithm keeps track
of whether the user is moving, or if they have come to a stop,
we can also support walk base.
So when you are -- when the user comes to a stop,
whether from a walk or a run, you'll get a pause event.
And as soon as the user starts to run or walk,
you'll get a resume event within your app.
Pedometer Events API, they look identical both
on iOS 10 and watchOS 3.
Let's take a look at the API itself.
You have the pause and resume events.
Each event is timestamped with the exact time
when the algorithm detected the transition from moving
to not moving state, and vice versa.
And you had to start and stop a pace to help your app register
and deregister for these events.
I've been talking about how pedometer events can be used
to clearly demarcate just the running segments during a
complete running workout and how you can, using that,
derive accurate metrics.
But you can also use pedometer events
in some other interesting ways.
Let's take a look at an example.
This is a trail-running app on the iPhone.
So here we are going to use pedometer events to see --
to figure out when we can engage with the user, and when we do,
how we can present them
with some interesting information that's going
to make them respond to those more actively.
And because this is a trail-running app,
there is going to be an elevation change throughout the
run so we are going to see
if there is some meaningful information there.
We are going to use the CMpedometer class to register
for the pause and resume events.
And we are going to get the relative altitude updates
or the elevation changes using the CMAltimeter class.
If you want to continue to receive pedometer events
within your app, even when the screen turns off,
your app has to stay running.
And one way of doing that on the iPhone is to subscribe
to continuous background location.
If you are interested in knowing more about this,
I suggest that you go check
out the Core Location Best Practices session
that is happening at WWDC today.
Next, we are going to register
for the relative altitude updates first
And then we provide an operations queue,
and provide a callback handler
to start receiving those updates.
In this example, I'm just going to make a note
of the most recent update.
But potentially in your app, you can cache all of those updates.
And at the end of the workout you can potentially provide a
nice elevation profile for the entire run.
Next, we'll register for the pedometer events themselves.
So first an availability check.
And then register by providing a callback handler.
This is just a quick tip to make sure that I don't run
into any concurrence issues, I'm doing work from all
of the handlers on the same operations queue.
So now that the app is set up to exactly figure
out when the user has started running,
and when they have stopped running.
We are ready to see if they can use
that in a very contextual manner.
As soon as they start running, we are going
to get the resume event in the app.
At this time you could make a note of that exact time
when we got the resume event, so that way at the end
of the workout, you can basically add up all
of those running times
to compute very accurate average pace for the entire run.
In this example, I'm just going to make a note
of the most recent elevation update,
so that I know exactly what elevation they started
when they started running.
And when the user comes to a stop,
now this is a very good opportunity for us
to figure out, because we have elevation, we can figure
out if they have run up a hill.
And if they have done so,
this might be their first ever hill run.
So why not just give them a hill bagging achievement.
Or if they have been doing that same hill run multiple times,
because we have exact, accurate timings for each of those runs,
now we can compare those and give them a stat
of how well they are doing on that particular hill run.
So that was an example where we use pedometer events not only
to arrive as very accurate metrics
and demarcate those running segments.
But it's also an example where you could use --
do something interesting with those events.
At the end of the run, of course you go ahead and pay
down the registrations and this will also release any block
handlers that have been captured in the framework.
Pedometer events are available on iPhone 6s and later iPhones,
and of course the Apple Watch.
So that's pedometer in iOS 10 and watchOS 3.
Next, let's talk about Device Motion.
As you're all aware, Device Motion APIs have been
on the iPhone since iOS 5.
With watchOS 3, now we are bringing the exact same APIs
to the Apple Watch.
With the Apple Watch, we have a very capable sensor package
that is at a fixed location on our wrist
and we use it throughout the day.
And because we use our hands for almost everything
that we do throughout the day, it's a really powerful tool
to observe all of that user activity.
And what Device Motion does is it takes the samples
from the accelerometer and the gyroscope, and it fuses it
to give you a very clear picture of all of that user activity,
and all of this right on your wrist.
All of that motion at wrist is described by Device Motion,
using four distinct properties: Attitude, gravity,
rotation rate, and user acceleration.
If you want to know in depth how these properties are derived,
and how they behave, I encourage you to go check
out these sessions from 2011 and 2012.
In this talk though, I'll be giving you a very brief overview
of this property so that we can build some infusion and go look
at some examples of how you can apply them in some health
and fitness apps in interesting ways.
The first property is attitude.
Using attitude in your apps, you can get an observation
of the orientation of device and space.
And when you use CMDeviceMotion,
you get these three representations of attitude.
As Quaternion, rotation matrix, and as Euler angles.
Whenever you are using attitude within your app,
one thing to note is every single attitude observation is
going to be relative to a fixed reference frame.
What this means is every single orientation observation is going
to be observed from a point
that is not fixed to the device itself.
So the reference frame remains static while the device can move
around, and that's how you observe
And furthermore, when your app registers
to start receiving the updates,
that is when the reference frame is set.
And so every subsequent sample that you receive
within your app is going to be computed using
that relative reference frame that was set
at the start of updates.
So this is something that you need to kind of be aware
of when you use attitude in your apps.
And so that you don't make assumptions
about where the device is oriented in absolute space.
The next property is gravity.
Gravity is -- well, it's gravity.
It is the force that is pulling us all
down to the center of the Earth.
And within Device Motion, it is represented
as a unit vector in the device frame.
Using gravity, you can observe the tip
and the tilt of the device.
But you might be wondering,
isn't different tilt also the orientation just like attitude?
But one key difference is, now you're observing all
of this orientation from the device frame.
So take for example, if I were
to hold my hand perfectly parallel to the ground
to my side, and then I move it to my front, you won't be able
to observe this using gravity.
Because the x, y, z components of gravity are going
to remain exactly same between these two orientations.
To observe something like this, you need a point of observation
that is external or fixed while the device is moving.
Which is basically attitude.
So you could use gravity to observe orientation,
but only in a limited form.
But it might work just good for the kind of app
that you are thinking of.
One other thing about gravity is
when you hold the device perfectly still,
you can observe it using the accelerometer.
It's a constant acceleration
that the accelerometer will pick up,
and you can read it right out of accelerometer.
But as soon as you start moving the device,
now the accelerometer is picking
up not just the gravity component, but it's also picking
up all of the user-generated motion.
So it becomes harder and harder
to just get the gravity component.
What Device Motion does, is by using sense of fusion,
we switch over to the gyroscope
to start tracking the gravity unit vector.
Next property is rotation rate.
As the name suggests, it's the rate
of change of angular motion.
It's very good for observing something like wrist rotation.
When you are rotating your wrist,
there is a rotational motion around the x-axis of the device,
and you can observe that using rotation rate.
You can also observe any rotation around the body.
Because most of us, we fix our torso and we move our arms
so there is going to be some kind of an arcing motion.
So there is rotation and you can observe it using the rotation
rate property of Device Motion.
The last of the properties is user acceleration.
And the user acceleration that you get
through Device Motion is compensated for gravity.
Recall how I mentioned that accelerometer picks
up both the gravity component
and the user-generated component.
So this is just the user-generated component.
But it has been compensated with gravity
that was derived using device sensor fusion.
So that was a very quick overview of these properties.
Now let's take a look at a few examples
of how you can use them in your apps.
The first property is attitude.
Attitude is very good
for observing any kind of repetitive motion.
Like rep counting in weightlifting.
So when you are lifting weights, you are literally going
through a set of orientation changes.
So just by looking at how those orientation changes are
repeating, you can count the reps.
And the best part about using attitude for doing
that is now they might be using a machine
where they're pulling horizontally or from the top
and you can observe all of those reps using attitude.
Gravity, as I already mentioned, it's very good
for measuring tip and tilt.
Which basically means you can use it in a yoga app.
So if they're doing a downward dog or holding a warrior pose,
then you can figure out whether -- how still they're holding it.
And when they're also going from one pose to another,
you can figure out whether they're really doing it
in a graceful way, or just falling over themselves.
One of the most useful ways of using rotation rate is
to observe the speed of circular motion.
Take, for example, a batting cage.
So when you are swinging the bat,
you're not only rotating your arms around your body,
but there is also the rotation around wrist.
So you could use both of those components of rotation rate
to estimate the bat speed.
Of course you'll need to know how long of a bat it is,
and at what point on the bat you want to measure the speed.
User acceleration is best used
when there is some abrupt motion.
Because accelerometer picks up all of the motion
that the user is doing, if you try to do some kind of app
where you're using user acceleration
to measure some small movements,
it might get drowned out by noise.
So wherever there is a very abrupt motion is the best place
to use user acceleration.
So something like a punch and recoil.
You can tell how much pain they're inflicting
on that sandbag, or that opponent
that is hopefully made out of air.
So those were only a very few set of examples
of how you can use Device Motion within your apps.
By -- after going through this,
if you feel that you have an app idea
that could use Device Motion and you want to find out more
about how you can apply Device Motion, please do stop
by our lab tomorrow, and we would be more
than happy to help you.
Now let's take a look at that API itself.
You have the four properties.
Attitude and gravity, they're unitlness.
Gravity's the uniflector.
User acceleration is in G's and rotation rate is
in radians per second.
Before you can start receiving Device Motion updates
within your app, you have to set their sample rate.
You can set sample rates up to 100 hertz.
And once you have set that sample rate, you can go ahead
and register for updates using the startup date method.
From this point onwards, you can either choose
to poll the Device Motion property periodically
to receive the most recent sample.
Or you could provide a callback handler
on which you can get every single update the Device Motion
is generating for you.
And of course, once you are done with the --
listening to the Device Motion updates, you can go ahead
and deregister using the stopDeviceMotionUpdates.
That brings me to the end of my section of the talk.
Now I'll be handing it forward to Paul, who will be walking you
through an example app on the Apple Watch
that uses Device Motion in an interesting manner.
Thank you very much.
Over to you Paul.
Thank you Bharath.
My name is Paul Thompson,
and I'm an engineer with Core Motion.
So Bharath just talked about what's new
with Core Motion APIs.
What I'd like to do is show you how to use one of them
in an Apple Watch fitness app.
So in this app, what we'd
like to do is create a tennis workout session.
Then we'll do -- we'll subscribe
to sensor updates with Device Motion.
Finally, we'll detect swings and differentiate
between a forehand and a backhand swing.
Now to do this, what we'll need to do is leverage capabilities
from Core Motion, as well
as some new capabilities from HealthKit.
Now as you may remember, watchOS 2 apps strongly relied
on HealthKit and Core Motion to do real-time analysis
and sensor output and provide you with values
such as step count, flights, distance, and calorie estimates.
Now, during a workout session, your app can do limited work
and process Device Motion samples while the user's not
directly engaged in your app.
But of course, this ability comes with some caveats.
To begin with, you must have enabled this new HealthKit
entitlement in your Xcode project.
Further, this background work can only be done during an
active workout session.
And in addition, while backgrounded,
you must minimize your CPU usage.
If you do not minimize your CPU usage,
your app will be suspended
until the user explicitly foregrounds it again.
Now since this background capability is being provided
by HealthKit, I encourage you to view the earlier session
to learn more in detail about this.
With that in mind, let's think
about what our app might look like.
Well with app, you would expect you and a friend to out
to the tennis courts with your Apple Watches,
and practicing volleying for a bit.
There, you simply start a quick workout,
and hit the ball back and forth.
Then, you'd expect that at any time you take a quick glance
at your watch, and get some media feedback on the play.
So with that in mind,
let's think about the structure of our project.
Here, there'll be three layers that we want to care about.
First is the UI, where we'll ultimately present
to the user what we've done.
Then, we'll have our workout manager.
We'll interact with HealthKit and start
and stop our workout sessions
and enable our background inability.
Then we'll enable our -- then we'll have our motion manager.
Here we'll interact with Core Motion directly,
and implement our detection algorithm.
There we'll respond to 50 hertz Device Motion updates,
and add the resulting samples to a running buffer.
Afterwards, on every update,
we'll assess whether a swing occurred, and if so,
we'll implement the UI.
So how do we want to -- how do we want to model the swing
that we wish to detect?
Well, tennis is a complicated sport.
So in this case, all we'll do is we'll look
at the essence of two gestures.
A full forehand and backhand swing.
We'll do this using the gravity and rotation vectors
as provided by Device Motion.
So if you expect the watch to be in the dominant arm,
then you would expect a full forehand swing
to include a simple rotation about the user.
So if we take the dot product of the rotation rate
from a potential swing with the gravity unit vector,
we can isolate this movement while ignoring the attitude
of the device.
And also ignoring some extraneous movement.
Then, once we've got enough samples of this,
we'll see if we've rotated far enough
and fast enough to count as a swing.
So now that we know what to do,
let's take a look at our sample app.
To begin, we'll envision our simple UI.
We'll display our information to the user.
Here we'll have three watch kit interface labels that we'll want
to update during runtime.
These will include the workout label.
The forehand count label.
And the backhand count label.
And what the workout label will do, will simply tell a user
when we've started and stopped the workout session.
And the forehand and backhand count labels will simply show
how many times we've detected the right movement.
Here we'll also have a force touch menu to start
and stop the workout session.
So now that we know what the UI will show, let's take a look
at our workout manager.
Here, we'll handle our interactions with HealthKit,
as well as create our workout session.
We'll also direct our motion manager below to start
and stop sensor updates.
So here, in our workout manager, we'll have to start a workout.
So to begin, we'll create our workout configuration
which we'll use to initiate the workout session.
Since we're creating a tennis app,
let's use tennis as the activity type.
And outdoors as the location.
Then, after initialization,
we'll have HealthKit start the workout session and subscribe
to Device Motion updates.
At this point, we'll now be able
to do work while the screen is off.
In addition, we'll also need to stop our workout.
Here, we'll simply do the reverse motion and unsubscribe
from sensor updates, and then tell HealthKit
to end the workout session.
At this point, normal backgrounding rules will apply.
So now, let's take a look at our motion manager.
He will interface with Core Motion directly,
and implement our detection algorithm.
So to begin, what we'll do here is we'll create a link
to the CM motion manager.
As well as create an NS operation queue for
or samples to do work on.
At this point, we'll also ask if the watch is
on the left or the right wrist.
There's a difference between the forehand
and the backhand swing will depend entirely
on this perspective.
We'll also keep a local count of a forehand and backhand swings,
as well as mark as whether we've recently seen a swing.
We'll also choose 50 hertz as our sample rate.
And create a running buffer
that should hold no more than a second's data.
Now as Bharath mentioned earlier,
Device Motion samples can be provided
at a rate up to 100 hertz.
While you generally want to pick the sample rate that's as small
as you can get away with, while also providing you
with fidelity that you need.
In addition, we'll set three constants which we'll use
in our detection algorithm.
These will include the minimum angle's [inaudible] swing,
the lower bound on the peak speed through the swing,
and an upper bound on the settling speed of the swing.
Now we chose these values based on experimentation
and sample data that we collected.
But generally, you'll find that the process of collecting
and picking these values will be half the battle of your app.
Finally, we'll create this delegate reference here,
which we'll use to communicate back to the UI.
So now, after we set all of our properties,
we'll adjust the operation queue to reflect as a serial queue
that we'll use to handle all of our Device Motion samples.
I'd like to emphasize here, that we chose --
we created this operation queue to ensure that all
of our data processing happens off of the main queue.
So now we'll also create this function which we'll use
to reset all the statement class,
as well as zero out the UI.
And then, as a final set of convenience methods,
we'll create some complementary update delegate functions.
Here, we'll simply implement our count of the swing,
mark that we've recently seen a swing,
and then let the UI know it.
So let's start interfacing with Core Motion.
So as always, the first thing we should do is ask whether the
API's supported on the device we're running on.
Then, we'll tell Core Motion to update us at a 50 hertz cadence.
And finally, we'll subscribe to Device Motion updates
by passing our operation queue, as well as a block
that we'll use to respond to all incoming Device Motion samples.
All this block will do is simply check to see
if there are any errors with the update, then pass it along
to our detection function.
So let's take a look at what our detection function's going
So as Bharath mentioned earlier,
Device Motion gives us quite a few things.
But in this example, we're only going to look at the gravity
and rotation rate vectors.
Now, as you may remember,
the gravity vector is simply Core Motion's destination.
The gravity univector,
regardless of how much the device has moved.
And the rotation rate is simply a rotation vector
for the device, giving us radians per second.
So now, what our [inaudible] will do is we'll take the dot
product -- will take the dot product of the rotation vector
from a potential swing with a gravity univector.
So we only analyzed the proportion
of motion about gravity.
Then, we'll add the resulting scaler
to a running buffer holding no more than a second's data.
Once we have enough content, we begin to analyze
in the content within.
So the two metrics we'll use
to analyze the swing are the accumulated angle of rotation
and the peak speed of the swing.
Here, to get the accumulating rotation simply integrate all
the accumulated samples that we've collected from rotation
over the second that we've collected them.
Then you have peak rate, you simply take a min or max,
depending on the direction of rotation.
Further down our function, we'll check to see
if we subtended far enough and fast enough to count as a swing.
If so, we'll choose forehand or backhand conveying the position
of the Apple Watch and the sign of the scalers.
And finally, to end the function, we'll add a check
to see that the swing is settled.
This way we can reject some of the recoil movement
as the user's arm moves back in position for the next volley.
And finally, to finish off the class,
we'll have the stopUpdates function.
Which we'll use to unsubscribe from DeviceMotionUpdates
when the workout has ended.
And that concludes the basics of our sample app.
We simply described a simple user interface.
We then created a workout management.
Handle interfacing with HealthKit.
And then we created our motion manager to handle Core Motion,
as well as implement our detection algorithm.
So I hope you all have gotten a good feel for how
to use Device Motion as newly brought to the Apple Watch.
So before I wrap up, I'd like to emphasize a few details
on the use of this API.
So now while you may expect the watch to be in a fixed
and predictable location, always remember to check
which wrist the device is on as this difference
of position can have significant impact on your analysis.
Further, when using inertial senses,
always try to remember --
try to understand what the reference frame you're
And further, as we said earlier, Device Motion provides you
with samples at a rate up to 100 hertz.
We always want to strike a balance
between the resources required to service your sample rate,
and the fidelity demanded by your application.
This is especially important,
given the restrictions placed during backgrounding
of a workout session.
And so to summarize the talk as a whole, we earlier talked
about the performance improvements
to historical accelerometer.
Then we demonstrated how you can use pedometer events
to precisely tag segments, provide important,
contextual feedback to users, and then we introduced you
to Device Motion on the Apple Watch.
And walked you through an app to detect forehand
and backhand swings during a tennis workout.
So now if you'd like to find out more information,
such as documentation and sample code,
please check out this link.
And further, as you saw, this app interacts with new features
from HealthKit so I encourage you all
to view this HealthKit session to learn more
in detail what's new for watchOS 3.
In addition, Core Location has some Best Practices for you
to review, and encourage you to check them out as well.
And finally, I also recommend checking out what else we have
in store in watchOS 3.
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.