Design for tomorrow's products today. See examples of how Apple and partners designed software for Apple Watch and other products before they even existed. Learn techniques for designing and prototyping so you can create great apps for hardware accessories and new products before they are available.
JEFFREY TRAER BERNSTEIN: Hello, everyone.
And welcome to designing for future hardware.
I'm Jeffrey Traer Bernstein.
With me are Matthaeus Krenn and Bill Lindmeier.
Who is this session for?
Perhaps you're creating a device that's connected
to an application.
That device could be, say, a dishwasher that you would
like to control from your application.
Or maybe it's a drone you would like to control from an app.
Or maybe a golf club that would send information
about your swing back to an application.
Perhaps you have an application and you would
like to create a device for it.
Maybe it's a game you would like to create a custom controller
for or a messaging application.
Or maybe a device to go with a dating app.
Who knows what that would be.
You might also find this useful if you are designing
for a device you just don't have yet like perhaps an Apple Watch
or some fancy new thing we've released.
Who are we?
We are a team of inventors, designers, prototypers.
We work across all of Apple's products
and explore what new interfaces mean to our devices,
operating systems and applications.
We brought you exciting features such as Multitouch gestures,
Force Touch and a Taptic Engine.
This time last year we did Prototyping at WWDC 2014.
Was anyone there for it?
[ Cheering ]
JEFFREY TRAER BERNSTEIN: Thank you - excellent.
Thank you for joining us.
Did anyone watch it online?
Less people, all right.
Well, it is not required for this session,
but I suggest you check it out afterwards.
I'll give you a brief synopsis right now.
We unveiled a disruptive,
revolutionary new application called Toast Modern
that allowed you to browse for artisanal, overpriced toast
on the streets of San Francisco,
take photos of the toast you're eating, and share it
with your friends and loved ones.
We used this application to explain why to Prototype.
There are two important reasons.
One, test ideas to save time
and money building the right things for your products.
Number two is to get new ideas to make the experience
of your product even better.
We also talked about how to Prototype.
Make fake apps, show people, and learn from their feedback,
and do it over and over again until the experience
of your product is incredible.
As you might have imagined, the reception
on the Internet was incredible.
People said things like,
"The workflow shown is something I may try."
JEFFREY TRAER BERNSTEIN:
The Prototyping video for WWDC is great.
"Grab a plate of toast."
" Takeaways: Number one, make fake apps; number two, um...
missed that part."
"I'm so proud of you."
So today, you're all very fortunate to be here.
We are going to unveil a brand new,
revolutionary Toast Modern product
that builds upon the success of Toast Modern, the application.
We are going to give you a peek behind the curtain, a little bit
of how Apple and partners prototype the watch.
We're going to create a device and an app that connects to it.
We're going to show you along the way a few prototyping
strategies for hardware and software.
I would like to share our Toast Modern product today.
Would you all like to see it?
JEFFREY TRAER BERNSTEIN: Excellent.
It's a revolutionary, disruptive, connected, social,
big data-y Internet of Things thing and this is it.
[ Music ]
JEFFREY TRAER BERNSTEIN: Thank you.
Thank you very much.
So we call this Toastal Service.
It is the world's first social toaster.
It allows you to send Toast messages
to your friends and loved ones.
Of course, it's available in three great colors: Silver,
Space Grey and 18-Karat gold.
JEFFREY TRAER BERNSTEIN: So we have an iPhone
and we have a Toaster.
Both have excellent qualities to recommend them.
The iPhone has a great interface.
It is connected to the Internet.
And it's always with you.
Our toaster can turn bread into toast.
So we want to combine the capabilities of each.
We want to blur the lines between hardware and software,
create a perfect meld of the two.
I can think of another product that has done the same.
That is the Apple Watch.
Now, you might think that at Apple we would begin here
with the fully formed Apple Watch springs from the brain
of its designers and is carved out of aluminum,
and sent to all of you.
In reality, we actually begin by drawing the Watch
on another device so we can experiment with the interface.
This is a pretty well formed watch as it exists today.
We start with this, a rectangle, and we use this platform
to experiment with the different things we might need to add
to the Watch to make the experience great.
We might add a Digital Crown that we draw on the screen
and scroll over to understand how that works and experiment
with different buttons.
This is great, teaches us a ton about the interface of the Watch
and the experience of using it.
But you can't strap an iPhone to your wrist and go outside
and try it in context.
So some of our app partners when they were developing
for the Watch before Watches were available
to them did some clever things.
For example, this French Transit app.
They created their own Apple Watch.
It is more rustic but useful.
They drew some pictures of their different screens.
And then slid them inside of the watch.
This was important.
This is a Transit app.
I need to be able to see what it's like to run
for the train I'm late for
and see what train I'm going to catch.
They have to try it in context.
The German newspaper Bild printed out a picture
of an Apple Watch and slid underneath various screens.
It was important for them to test the legibility
of their articles in the places
where they expect people to read them.
We learned a few things
about Prototyping hardware and software.
We learned we can fake hardware on screens
as with the Apple Watch.
We can fake software with pictures
as we explained last year with Toast Modern,
and it is important to try it in context at the right size,
in the right place as you saw with the paper Watch Prototypes.
Here is our prototyping cycle from last year:
make fake things, show them to people
and learn from their feedback.
This year we are going to make fake hardware and software.
Before we begin we are going
to ask ourselves three important questions for each of those.
What needs to be more real?
What can be faked?
That's nearly everything except for the one thing we are going
to choose to be real, and where will it be used?
And then we are going to go and show it to people.
Who do we show it to?
The people you app is for.
If it's not meant for your office mate,
go find the people it's for.
Where? In the place where people will use it.
You might have to go outside.
And remember, don't argue, defend, or dismiss.
You're there to learn from people's impression
of the Prototype you put together.
After we've shown it to people, we learn from their feedback.
We ask ourselves three questions:
Number one, what is working?
Number two, what is not working?
Number 3, what other ideas does this give us?
Now we are going to go through the cycle three times today
and show you a few different techniques.
We are going to begin with pictures and animation.
Then we are going to use a technique we call behind
After that we are going to create a lightweight interactive
and connected prototype.
I would like to quickly recap Toastal Service for you guys.
It's the world's first social toaster.
Allows you to receive toast messages, we call them Toasties,
and send toast messages,
and we want that experience to be magical.
To kick it off, I'd like to ask Matthaeus up to the stage.
MATTHAEUS KRENN: Thanks, Jeff.
Cool. So how are we going to get started with this?
So first of all, let's look at our hardware and our app
and ask ourselves the question that Jeff just mentioned.
First of all, what do we make more real now?
Well, I want us actually to start with the receiving
and the Toasting part of the experience.
So on the hardware, let's figure out a way how we can indicate
that a new Toastie has been received.
On the app we can look at new Toastie notifications
and see what kind of information the app can display with it.
The next question is, what can be faked?
And really here just go for as much as possible really.
We are not going to build a Toaster.
We are not going to look at how to turn bread into Toast.
All we are going to do is put pictures on the device
and simulate the hardware like that.
For the app again, everything we are going
to take pictures for everything.
We are not going to make anything really interactive
and definitely not writing any code.
And third, where will the product be used?
For the Toaster, that's the kitchen.
For the app it's anywhere or at least anywhere
around the house for sure.
So let's start with looking at our Toaster.
We said that we wanted to simulate hardware
by putting pictures on a screen.
That sounds awfully a lot like what Jeff just described
with the Apple Watch on an iPhone.
Luckily Apple also makes a great Toaster shaped product called
That is just the right thing for us here.
To sketch out our Toaster, I'm going to use keynote today.
For you really, whatever gets you started easily and simply,
I'm going to use keynote today.
Feel free to use whatever you can.
I'm going to try to keep it as simple as possible too.
A Toaster is really just a box, some feet, a lever,
maybe we will actually call this a digital lever.
And a slot where you can put in a piece of bread.
We also said that we wanted to indicate
when a new Toastie has been received.
We'll put an LED up there that can light up.
And while we are in keynote, we are also going
to add some bread to the Toaster.
This is the Toaster toasting.
And eventually the Toastie pops back up
and we can see what's on it.
We can take this.
We can open it in keynote on our iPad.
And we have a Toaster.
For our app we are going to do something similar.
We are going to get started with a home screen, a screen shot
of a home screen on the iPhone and put our app icon on it.
We are going to take a screen shot and use it as an underlay
for an application that we want to customize.
For our app, we're going to take a screen shot of Toast Modern,
the product that we unveiled last year.
You can see in the top left corner there's a convenient open
spot that is the perfect place for the link to our Toast inbox.
I'll put a Toaster icon up there and a little notification badge.
On the Toaster inbox I'm going to use a slice
of bread right now as a place holder because I don't want
to take away what is actually on the Toastie.
We don't want to take away the anticipation,
that surprise moment when you are sitting waiting
for your Toast to be toasted.
We definitely want to show who is this Toastie from?
We will put a picture and name of the sender up there.
And we are also going to add the date and time when it was sent.
At this point I think it would be great
if we can immediately also choose how dark we want our
bread to be Toasted.
We can do that easily with a slider here.
Now, again let's open that on our iPhone and see what it looks
like on an actual device.
This immediately gives us a really life like
but completely fake app.
Now we can look at how these two things work together in action.
You can see that I'm just tapping on a screen
to make the notification appear to open the app
and to go into the inbox.
And on the Toaster, I can also -- you can first of all you see
that the LED is already blinking.
Then I can tap to insert some bread.
I tap again to lower our digital lever.
Then in a couple seconds later the Toastie will pop back out,
and we're going to see what's on it.
That is not too bad for now.
We can immediately take this and show it to our friends
and ask them how they feel about this so we can learn
from what they have to say.
So what is working right now?
Well, the LED seems to be really good indicator
that there is a new Toastie, and the notification
on the app helps us know that there is a new Toastie even
when we are not sitting in the kitchen
in full anticipation looking at our Toaster.
What is not working so well is, a lot of people mentioned
to us what happens when you have more than one Toastie?
That's true, our design doesn't account for that very well.
So what can we do about this?
Well, maybe we'll actually upgrade the LED on our Toaster
and maybe our app can show a list
of Toasties instead of just the last one.
So let's take it from the top here again.
Let's look again at our hardware and at our app.
And at this stage again let's ask ourselves what needs
to be more real now.
Well, we want our hardware to handle multiple Toasties
and toast the right one.
On the app we want to be able to browse Toasties
and select the right one for us.
What can we fake?
Still we are going to fake as much as possible.
There is not going to be a real connection
between the two devices.
Anything related to actual toasting we are not even going
to touch that.
For the app sending data back and forth, again we are going
to fake that and our UI is mostly still pictures
Where will the products be used?
Still the same as before.
The Toaster in the kitchen and the app really anywhere.
Let's look at our Toaster again.
We said we wanted to change something about the LED.
So maybe we can use a dot matrix display here instead.
We could indicate how many Toasties have been received.
And instead of using hi res touch screen that we put
on the Toaster here which would make our product complicated
to produce and expensive to buy, we are not going
to recreate capabilities that we have on the phone anyway.
We'll keep the hardware simple and focus
on the complicated stuff on the phone.
So we'll add additional Toasties to our inbox.
Easy to swipe through so we can select
which one we want to Toast.
We will create a state where a different Toastie has
We will also add a screen that shows up on the app
when the Toaster is toasting.
We can add a count down to the top
so we know how long it will be
until we enjoy our delicious piece of toast.
Finally the Toastie is toasted and it's back on the list
and we can see what's on our app as well.
Now we have an iPhone, now we have an iPhone running a keynote
of a fake app and we have an iPad running a keynote
of a fake Toaster.
With those two together we can start playing more realistic
scenario of our product experience.
A notification comes in on both devices at the same time.
We open the app.
We go to our inbox and select a different Toastie.
We add some bread to the Toaster,
push down the digital lever
and eventually the Toastie pops back out on both devices.
Now, that's pretty cool.
But there is a fundamental problem with this unfortunately.
When we test our product, we can't just put the devices next
to each other in front of us and happily tap on both of them
in a really orchestrated way to play this perfect scenario.
What is more realistic is that eventually the Toaster is going
to be in the kitchen and the app is going to be all
over the place with you.
That's a fundamental part of the experience, right?
You get a notification.
You walk up to the Toaster.
You do a couple things.
It starts Toasting and you walk away
to brush your teeth, something like that.
Eventually the Toastie is done
and you come back to the Toaster.
If you don't capture that part of the experience,
you're testing for something that is entirely unrealistic.
How can we make our two Prototypes talk
to each other over a distance.
Ideally how can we do
that without putting any additional effort into it.
I want to keep using the keynote prototypes we just built.
It turns out our team has a really convenient technique
for circumstances like that.
I want to show that to you right now.
You can see this is still our keynote prototype on the phone.
But now the notification actually comes in on its own.
We can still tap to open the app and then go
and select the different Toastie that we want to Toast.
And on the Toaster, you can see
that we can still tap to insert some bread.
Now look at the app, what happens when we lower the lever.
The app responds to it in realtime.
When the Toastie is back finished Toasting,
it appears on both devices at the same time.
That's pretty awesome, I think.
But how do we do that?
Especially considering that this is really just images
on devices, and we didn't write a single line of code.
Well, it turns out we have something
that is a little bit cooler and a little bit faster
and a little bit smarter than code.
MATTHAEUS KRENN: We call this ...
MATTHAEUS KRENN: We call this technique behind the curtain.
It is essentially about a person pretending to be a computer.
In our case, it's me behind the sofa,
using two keynote remotes connected to the iPhone and iPad
and I can supplement tapping on the Prototypes
by remotely controlling the state of each device.
When you do something on the app I can press on the remote
and the Toaster reacts to it.
We create a perfect simulation of this realistic experience
and we didn't really have to do anything to achieve it.
You wouldn't believe actually how many people
at Apple we tricked
into thinking we have implemented some magical
technology where really, as we were demoing our Prototype,
some of our team is just kind of sitting
in the corner controlling one of the Prototypes.
And so again with this more realistic setup we can go
to our friends and ask them for feedback.
And we ask them what's working.
Well, selecting from many Toasties
and then Toasting the right one, it's all done, all good.
What is not working?
Well, despite our incredible Prototyping efforts,
the devices still don't feel really connected and responsive.
What can we do about that?
Maybe we can have the Toaster react in realtime to the input
in the app and vice versa.
To do that we are going to have to step
up our Prototyping game a little bit.
I'm going to ask Bill on the stage
to show you guys how to do that.
BILL: Thank you, Matthaeus, that's awesome.
Hello, everybody, I'm Bill.
I get to spend my days experimenting
with top-secret Apple hardware which is pretty awesome.
So we would like to share a little bit about our process
that might give you some ideas for designing
for your own top-secret hardware or possibly a device
that you just don't have yet.
So after a couple of iterations, we are feeling pretty good
about the designs that Matthaeus created for us.
However, with the keynote prototypes,
it can only take us so far.
We can tap through from state to state or fake that,
but there is no true interactivity yet.
For example, the digital lever.
We can't drag that up and down, which is pretty critical.
So let's return to our three questions
and decide how can we enhance the fidelity of these prototypes
with a little bit of interactivity and a little bit
of connectivity to make them that much better.
What needs to be more real?
On the hardware side we want to wire up the digital lever.
We also want to connect the state
of the Toaster to the app itself.
On the app side let's allow our friends
to browse the Toasties in the inbox.
Also pick a Toastie to Toast.
So we like faking things.
What can we keep fake?
Let's not worry about servers, edge cases,
turning bread into Toast.
And on the app side, we are going to keep
that message queue completely canned
and also the Toasties themselves,
we'll have them be images on both devices.
We won't actually send them back and forth
between the devices over the air.
Okay. So where are we going to use these things?
Toasters live in the kitchen,
so we'll use the Toaster in the kitchen.
The app can be used anywhere
that maybe you're hungry for a snack.
Maybe it's the first app you launch when you roll out of bed.
We have to ask ourselves,
what is the best way of moving forward?
Matthaeus said he wanted to connect the Toaster and the app.
Maybe we start with a real Toaster, hack it together
with a microcontroller.
Maybe put a wireless radio on it, some LEDs to make
that a nice little display.
And this would probably work.
But we are not committed enough to this design to move forward
in this direction because this is starting to feel very real.
Maybe it's time to 3D print a model of our Toaster
and start customizing Toasties with a laser cutter.
Let's be honest, we are always looking for excuses
to use the 3D printer and the laser cutter.
BILL LINDMEIER: But it is actually really fun making this
slide, but that is probably putting the cart
in front of the horse a bit.
Okay. So how about a virtual reality Toaster?
[ Laughter and Applause ]
BILL LINDMEIER: This will be extremely awesome and fun
to use, but Toaster haptics are still about five years out,
so we have to put it on hold.
Seriously, all three examples have in common
that they are distracting us from the task at hand
which is designing the experience.
So to the extent that we do any engineering at this point,
let's keep it to the engineering which enhances our design
and lets us explore that a little bit further.
That way when we actually do go to make this thing we know
that we are making the right thing.
Okay. So with that in mind, for the purposes
of our Prototype, what is a Toaster?
It is not a bread heating device, but you knew that.
It is a connected state machine.
When I say that I just mean that it's a physical object
with attributes like where the digital lever is, whether
or not there's Toast in it, what message is toasting,
and we want to know how do these attributes respond
to user input.
We can easily model this with software.
So we are going to take our existing keynote Prototypes,
the app of the Toaster.
We are going to port them to Swift.
And when I say we are going to port them to Swift, I don't mean
that we are going to make a fully functional,
feature-rich Swift app.
It is a lot like the Prototypes in that there are screen shots
and they animate based on user touches, but then we are going
to sprinkle in just a little bit of extra code
for features we want to explore further.
Of course this doesn't have to be Swift.
It can be another programming language you like.
It can be another tool that allows you more interactivity.
But we at Apple happen to like Swift so we are going
to move forward with that.
Okay. So specifically let's define what we are going
The Toast-a-Toastie flow.
So for the first step we want to pick a Toastie from the inbox.
Next we want to send the Toastie to the Toaster.
Third, we want to Toast the selected Toastie and finally,
we want to indicate that the Toastie is Toasted.
So here is our new interactive app prototype.
What we've done is turned this carousel into a UI scroll view
that allows our friends to easily paginate
between the different messages.
We have a UI slider at the bottom which allows us
to pick the color of the Toast.
We've also added a gesture recognizer to each slice
of Toast so we can pull it down into the virtual Toaster
and simulate that it's being sent.
Let's see how that looks in action.
We launch our inbox.
You can see as we swipe through the Toastie selected increases.
And we can adjust the Toast darkness variable.
And then we are going to drag the Toast
down into the virtual Toaster which sets Toasting to true,
kicks off an animation and our Toastie is presented.
Moving on to the Toaster.
We are going to add a gesture recognizer to the digital lever.
When you pull it all the way down, it begins Toasting,
set Toasting to true, begin a timer.
We have also added a segmented control at the bottom
that lets us easily pick the message we want to Toast
without connecting it to the app,
and finally the animation will present the Toast.
Let's take a look at that.
We are selecting our Toast which loads the bread.
You can see we have this nice animation now in the LED matrix.
When we bring the lever all the way down to 1,
it is Toasting set to true, kicks off the timer.
And our Toast is presented.
So this is a critical stage.
This technique of modeling hardware
with software is super useful and we use it
in our studio all the time.
You can imagine that you are making a connected Fireplace
and you want to integrate that with HomeKit.
You can use a projector or that large screen TV that you've got.
Put it where you expect the Fireplace to be in the den
and you can simulate the controls right
where they are going to be used.
Maybe you are creating the next great Transit app for the Watch
but you don't have a Watch yet.
Throw the interface on an iPod, bring the iPod
down into the subway station and use it in context.
Maybe you're making the next great drone.
You can simulate the hardware controls on an iPhone,
strap it to the paper prototype,
and when you're moving it around, you can stream gyroscope
and accelerometer data back as if it's the real hardware.
So we've got the interactive app, the interactive Toaster.
Now we need to send data across.
Pick the Toastie from the app and send physical states
across from the Toaster back to the app.
What is the best way to do this?
The best way is whatever is easiest and fastest for you.
We are not designing a network API.
We can let somebody else do that later.
We are not worried about efficiency.
If you're Prototyping in the browser,
maybe you use Web sockets.
If you've got that drone and the phone connected to it,
you can stream the gyroscope data
with O.S.C. Maybe you have a back end server
and you can use HTTP requests.
I know that some of you have probably never programmed a
networking layer before.
So think creatively.
You can use a service like if this then that to observe emails
that you're sending back and forth between the devices
and take action upon that.
Maybe you're noticing whenever a new photo is added
to a shared photo stream.
So we are going to use a Multipeer connectivity peer
to peer connection.
This is what our network is going to look like.
We have two iPhones connected to an iPad
and call our network Toast Talk.
The reason we're using Multipeer connectivity,
it is incredibly easy to set up.
It doesn't require a server and it allows an arbitrary number
of devices to connect.
So here is a bit of code.
I'm going to fly right through because we are running short
We are going to create a session.
We are going to name the network Toast Talk and take any data
that comes in and convert it to a string.
We can choose devices with this handy class called the MC
Browser View Controller.
Super easy to set up.
You pass the name into the constructtor
and also pass in that session.
You can present it on a button click or when the app launches.
Then you just dismiss it when they've connected.
What messages should we send back and forth?
The app Prototype wants to tell the Toaster the Toasty number
and the Toast darkness.
The Toaster needs to send the lever position
when Toasting began and when Toasting ended.
A little bit of code that does that.
We are just going to convert our string to data
and send that to the session.
And this is an example of how we might send the letter position.
We are going to say we've got our message
which is lever position.
Throw a carat in there and pass some data along with it.
This is a totally arbitrary format but fast and simple.
We will split the incoming messages on that carat
and switch overall the message names and pass the data off
to whatever function is appropriate.
This allows us this really cool dynamic relationship
between our Toaster and our app.
You can see as I drag the digital lever up and down,
that's reflected in the app.
That's pretty great.
We've closed the loop.
We have interactive apps and Toasters.
The world is good.
Let's give this to our friends, let them try it in context
and see what they think.
[ Music and Applause ]
BILL LINDMEIER: So Toastal Service is shaping up
and starting to feel real and we haven't done
that much engineering.
Let's learn from the feedback.
What is working?
Clearly we can pick messages from the inbox
and send them to the Toaster.
That's a fundamental part of this experience.
What is not working?
Our friends told us it's a little odd
that there's no visual feedback from the Toaster
when you select the color from the color slider.
Maybe in the next round
of Prototyping iteration we adjust the color of the LEDs
on that matrix to reflect that the data is being sent over
and the color is being remembered.
So with that, I would like to invite Jeff back up to the stage
and he can wrap it up for us.
Thanks for taking so much time
on that last sentence, all right.
Quick recap for everybody.
We Prototyped using pictures and animation
so we could quickly see different inputs
and outputs for the Toaster.
We showed you the behind the curtain technique
to see how the app and the device work together.
Then we created a lightweight interactive
and connective Prototype
so we could try the entire end-to-end experience.
Along the way we learned a few things about our Toaster.
We don't need a darkness setting on the Toaster
where most Toasters have that.
We need a number display on the Toaster.
We need a sensor on the digital lever
to communicate back to the app.
We figured all of this out really, really fast.
Look at all the time and money we saved.
Imagine that we had to build a real Toaster to learn all
of these things and we got really cool ideas along the way.
So all of that reminds me:
There are two important reasons to Prototype.
Number one is to test ideas to save time
and money building the right things for your app.
Number two is to get new ideas to make the experience
of your product even better.
How do we do it?
We make fake things.
We show people and learn from their feedback.
And we do it over and over again until our product is incredible.
So here is what we would like you to go and do after this.
If you are creating a thing connected to an app,
for the thing, fake it on a screen.
For your app, fake it with pictures in context
and for the connection between the two, just tap through both.
Or use the behind the curtain technique or behind the sofa.
And maybe create a lightweight network prototype.
If you just don't have the device,
especially perhaps the next thing
that we unveil, don't wait.
Fake it on a screen, in context.
Here is some stuff about more information.
There are related sessions.
Most importantly, tomorrow Designing with Animation.
Some of our teammates at 3:30 in this very room.
I think you'll find it very useful.
Thank you very much, everybody!
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.