Designing Sound

Hello.

Welcome, everyone.

Imagine, the next version of your app will have no graphical UI at all.

How would you interact with it? What could that sound like? I'm Hugo Verweij, sound designer in the Human Interface team. I had the pleasure on working of many of the sounds you hear through your Apple devices, and I love thinking about these kind of questions.

There are probably some musicians in the room, maybe even one or two sound designers, but that is by no means required for this talk.

This session is for everyone, even if you've never recorded a single sound in your life.

I'm also not going to teach you how to become a sound designer, but I hope I'm going to give you some ideas of how to incorporate sound in your design process and why it makes sense to do so. Now, as a developer, you have a lot on your plate and adding sound to your app is probably not super high on your to-do list, but I like to make the case that sound, if used the right way, really can help transform the experience of your product. Now at Apple and at WWDC, we talk a lot about visual design.

For now though, let's just try to use our ears for a little while. I'd like to ask you all to just close your eyes for one minute, and imagine you're in a forest.

This might help you.

It's a quiet afternoon, and you're going for a walk. The weather seems very nice. But then [phone notification sound] Dark Sky is predicting rain.

You better pick up the pace.

Okay, you can open your eyes now.

Oh, we're not in a forest. We're actually at WWDC. Sound has this magical ability to calm us down, to get on our nerves, to scare us, or to delight us. When we close our eyes, it has the power to transport us to places.

I want to talk about the following things today.

Why sound is an important aspect of design, designing sounds for notifications, designing sounds for UI elements, and some tips and techniques to get you started. Sound is everywhere, all around us, and we never really pay much attention to it. Church bells have been telling the time for many centuries in countries all across the world, like here in my beautiful home town, Amsterdam. They tell us the time. They tell us where we are. They give us a sense of context.

We also use our ears for an early warning system, like when we're about to cross the street and we hear this -- [brakes screeching] -- we don't need our eyes to understand it's not safe to cross. Or we listen for quick confirmations every day. When you put on your seatbelt , you don't have to double check it's fastened. You hear the click, and you know you're good to go. We can also listen for more nuanced things, like the sound of someone's footsteps may tell us something about their emotional state .

Are they calm? Are they in a hurry? And if we know a person very well, we may even recognize them by the sound of their footsteps. . So what we hear has a big impact on how we feel and what we do, and sound helps us make sense of the things going on around us.

Now we know that sound is such a powerful medium. How can we use it in our design? Many people think that design is about what a product looks like or at Apple we pay a lot of attention to, how it works.

We think design is also about what it feels like and what it sounds like.

So what is the role of sound in design? It's like using a universal language that is already understood by everyone. It helps shape the experience of a product. It helps us understand interactions better. And it gives our devices a voice so they can tell us things, they can talk back to us.

So if your iPhone is playing a sound, what is it trying to tell you? . You just used Apple Pay successfully. . Or maybe it didn't go so well. Try again, please.

. Somebody just texted you.

Or good morning. It's time to wake up.

Or at the end of the day it's almost time to go to bed.

Many times though, when an app plays a sound, we don't really know what it means. It sounds like this. . We have to look at our screen to understand what's going on. It's like a way for our phone to say, "Hey, look at me. Something happened, but I don't really know what." It's not a very intelligent way of communicating. So how can we improve on this? Well, if your app relies on notifications, it makes a lot of sense to think about creating your own notification sound. What do you want people to experience when they receive that notification? Do you want people to recognize that it's coming from your app? If you don't do this, you are stuck with the default we just heard, and it doesn't really say much about where it's coming from and what it means, so it's a missed opportunity to distinguish your app, your app's unique identity. Now, before, when we were in the forest, we heard the Dark Sky notification, and Dark Sky is a weather app, and it sends this notification when rain is imminent.

It does this pretty well. Let's listen to it again. And why does this work so well? . This sound is easily recognizable. When you hear this, you know it's coming from Dark Sky.

It conveys a message. It's not a direct recording of raindrops, but it has this liquid quality to it. It helps you connect the sound to the message it's sending. It's a friendly sound. It's not jarring, but telling you in a friendly way it's time to find cover soon.

And it's a simple and clean sound, quite pleasant to listen to as well. So let's try this ourselves. Let's create a sound for an app. Shall we? Well, first we need an app, of course. Well, luckily, some years ago, our prototyping team did a session. It was called "Fake It Till You Make It." It was about creating fake apps in order to learn from how people would interact with them.

They create an app called Toast Modern.

Now, maybe some of you know that toast has become very popular in San Francisco, and people pay a lot of money for a slice of what some call hipster toast.

Toast Modern, the app they created, helps you find overpriced artisanal toast on the streets of San Francisco and share this with your friends and family. Now if someone sends you some newly discovered toast, you get a notification. So we can create a sound for this notification. Let's do that.

We can start by asking ourselves some questions, like, "What is the essence of what this app is about?" Well, in this case, simple. It's about finding toast and sharing it with your friends. What emotion do we want to evoke here when people receive this notification? Well, we want them to be happy they found some new toast to try and eager to try it out.

How do we match the app's aesthetics? We just saw the screen, and it looks pretty functional and simple, so let's keep the sound similar to that, keep it simple and to the point. And if we notice, we can start with some sound sources. We can record some sounds to be inspired by. In this case, let's just get some bread. Let's get a toaster and record those sounds.

So that's exactly what we did, and here are some of the sounds we recorded.

Toaster ejecting toast.

Spreading butter on a piece of toast. The sound of toast breaking.

And taking a bite out of a piece of toast. Now we can also create a little bit of a story from the beginning to the end, you know, making toast and eating toast, and record the whole process. . . . . . . Okay, now we have a nice palette. You notice we added some sounds to this, because in the initial set, there was not that timer sound and the chime sounds, so we recorded those two. And now we can start making notification sounds. There's one sound that I like quite a bit. . It's a great sound. It's very characteristic, but maybe it doesn't work so well as a notification coming from your phone every day, right? It's a little harsh. So how about this one? . . There's this little timer sound and then the chime, but the only thing I don't really like about it is the sound of the toaster ejecting the toast. It's a little harsh. So we can find a more expensive toaster maybe, that'll sound better, or we can construct our own toaster sound.

But why don't we just take away the parts we don't like and see what we're left with? So let's take that sound out. . . Actually, this might work pretty well. Let's see what it looks like when the notification comes in.

. . I think this is pretty good for now. Now remember, we're just trying things out. We're just prototyping. Even the app is fake. So, you know, we're not all sound designers, but once we start working with a sound designer, they can take this inspiration and maybe create something that's more unique, more ownable, and maybe more abstract, like the Dark Sky notification.

Now what did the creative process look like here? We started out with all these toast recordings. Those are the blue dots. And then we realized we needed some other sounds to complete the palette, so we added the timer sound and the chime sound. Those are the yellow ones. And then we created some candidates and tried them out on our device.

We found out that a lot of these sounds didn't really work so well, so we ended up with the timer and the chime sound.

Now why am I showing you this? Because it illustrates how often we start out with this assumption that something might work, but then we find out that, going this exploration phase, we find out that what we really need is something different. We have to go through this to understand that. First we have to make mistakes to come up with something great. Now what kind of sound sources can you record when you do an exploration like this? You could record an acoustic instrument or an electronic instrument, like a synthesizer software or hardware.

But really, we found out you can record anything.

Anything that makes a sound can be an inspiration. Now at Apple, we use any of these sound sources. It really depends on the project we're working on, and some of them are easily recognizable instruments.

These sounds may be played millions of times per day.

Earlier this year, Tim mentioned that at one point during the Super Bowl, 380,000 messages were sent per second.

Never though in the history of the sound has it been played live on stage.

Really, nobody ever thought of that.

But today's a special today.

So ladies and gentlemen, for the first time in the history of this sound since it was introduced with iOS 7.

Here it is. . Concentration.

I hope I hit the right one.

.

Thank you very much. We use this notification sound for messages. Why does this work well for messages? We have to think about, when will people hear this sound, and what will they be doing? They have their phones in a pocket, in their jacket. Maybe they are on a busy city street or an airport. So we have to cut through the noise of these locations. It needs to be very clean and simple, but there's another thing that is important that we don't really want to, you know, convey the information of what you're sending. It's very different than the Dark Sky notification that can only mean one thing. It's going to rain. In this case, it can mean anything, anything you're sending with it. So we want it to be very neutral as well.

Now this here is a kalimba. Maybe you've seen one before. This is what it sounds like. You recognize it? Maybe you do now. . So we recorded this sound as the default notification for -- anyone? Calendar, that's right. Absolutely.

It's a little bit more of a characteristic sound, because it's not a very common instrument, but again, it's very simple and clear as well.

Now when we were designing for the watch, we asked our self the question, how can we be inspired by this beautiful industrial design of this device? And how can we create the illusion that what you hear, that the sounds are actually made by the device itself, that they're coming from the device itself? So we decided to record the different materials, the stainless steel, the aluminum, and the gold.

And what I have here is the housing of a stainless steel Apple watch. Now we can hit that in different ways and see if it makes a sound. . If I hang it up, try a couple times. . Hit it the right way. . It actually sounds quite nice. So we decided to use this as an inspiration, and we used these actual recordings for some of the sounds that are on your watch today, like the notification and the watch ring tone.

So together with haptics, what you feel, what you see, and what you hear really come together here in one experience.

Now you may still think, "I have no idea how to record a sound. I've never done that before. Well we've got some very simple tools you can start with, already on your phone. There's Voice Memos. It's a great recording app. You can record any kind of sound with that. And there's Music Memos. You can download it for free from the app store, and it's a great tool for more musical ideas, and it works really well with Garage Band. Speaking of which, Garage Band can be used on your Mac to record sounds and edit sounds.

And for those who want to dive a little deeper, there's always Logic Pro, a great tool for the more seasoned users.

So to recap, what are the characteristics of a good notification sound? It's distinguishable.

You know which app is sending the sound.

It matches the design of the app. So you're creating an app that looks very cartoony, let's say. You may want to look at some sounds that really match that aesthetic. Or if your app is very minimalist, then maybe you want to start with some very simple sounds.

It's unobtrusive and repeatable. This means that even if you use it multiple times per day that it doesn't become annoying.

A great way to try this out is just to put it on your phone for a little while, for a week or so, and live with it for a while.

If after a week you still like the sound, it's probably a pretty good one.

And then it's clear. It's clean.

It cuts through noise, but it's not abrasive.

All right. Next, let's look at designing sound for UI elements.

Now every object you interact with in the real world makes a sound, so it's very natural to expect that an object you interact with on the screen makes a sound as well, but we definitely don't do this for every interaction.

We really try to find the right balance between no sound and too much.

So UI sounds are things like keyboard sounds, scrolling sounds, pressing a button, tapping, swiping, or any kind of transitions. Now I want to focus on three of these and see how we approach them in our own devices. Let's start with the keyboard sound.

Now following our previous design strategy, we may want to start by recording some actual keyboards, like an old typewriter.

Now, this rattles quite a bit, but there's something satisfying about the letters hitting the paper. So let's remember that.

How about a computer keyboard? There's something pleasant about this sound as well. It's not very static. It's very lively. So not every key sounds the same.

So we can remember this when we create our own sound for our iOS keyboard.

We're designing for software, so we can make it sound any way we want. For instance, we can change the sound of the space bar, of the modifier keys, and the backspace keys.

Let's listen to it. So we create a character that is a bit more playful and not so static.

And we do another thing. When you type faster, we bring down the volume of the sound ever so slightly.

It's not something that you may hear in everyday use, but even these little things make a difference in the experience. Let's listen to it one more time.

Okay, now next it's pressing a button. For those of you with an iPhone 7, I invite you to bring it up to your ear and click the Home button a couple of times. Go ahead.

What does it sound like? Now since the iPhone 7, we have a solid, safe Home button. It's durable, and it's waterproof, but it doesn't actually move or make a "click" sound. So we used the Taptic Engine to create this button experience, and we learned while designing for the watch that combining sounds and haptics the right way and synchronizing them precisely, we can come up with this great multisensory experience, and we were looking for a similar experience for this button, so we decided to add a very subtle, quiet "click" sound to the tap of the Taptic Engine, resulting in what you hear right now when you press it. And this also allowed us to create multiple "click" fields. So when you set up a new iPhone, you can choose between two different clicks.

There's one more thing that was very interesting when we were doing this exploration.

We found out that, if you take one haptic tap, and you synthesize it to a very sharp and precise sound, like the one on the left here. It sounds like this. And people think that what they feel is very precise and sharp as well, but if we take exactly the same tap, exactly the same, we didn't change anything, and synchronize it to another sound like this , then people think that what they feel is very different.

It's less sharp and less precise. So it's really interesting how we can fool ourselves using sound this way.

And lastly, let's look at transitions.

These are the sounds that accompany a movement between two different states, like in the Apple TV. We're looking for two things here. A feeling of depth and a feeling of movement.

Let's listen to it.

So the special character of this sound emphasizes the feeling of depth, and the envelope of the sound is shaped in a way similar to that of an object passing you by, creating that feeling of movement. We could have used another sound with similar characteristics, but you really also want it to match the aesthetic of the Apple TV user interface perfectly. We think this sound does the job really well. And there's also a sense of direction, because we have two different sounds. You know if you're moving forward or backward. Let's listen to it one more time. So what are the characteristics of good UI sound design? First off, it's used sparingly. Really, again, try to find a balance between no sound and too much. It's a delicate balance. It has a much lower volume than a notification sound, because people have their phones in their hands.

We can keep it very quiet. We only want to add a very subtle layer to the experience.

Now they help the users, and they delight the user by creating this great visceral experience combined with haptics and animation.

Now in this last section, I'd like to give you some tips and techniques to keep in mind when you're working on your own app.

First of all, keep sound in mind from the start.

When you're beginning your project, you can ask yourself these questions. Will my app send frequent notifications? If this is the case, it makes a lot of sense to think about creating your own notification sound.

Can sound play a role in my app's branding? Well, if the answer to the first question is yes, and you do create your own sound, then it's already playing a role. You're already creating a voice for your brand.

Now how can the UI benefit from an audible component? Is there any way you can help people understand what's going on on the screen by adding some subtle sounds to it? And then how would I understand my app without a graphical user interface? The question I started out with. Of course, you don't have to use your app without looking at it, but it's an interesting question to help yourself, and it may help you come up with some innovative ideas. This is an important one. Silence is golden. It may sound funny from a sound designer, but the absence of sound is as important as adding sound.

Really try to find the right balance. If we wouldn't do this, our Maps app could easily sound like something like this. 3D. A bit much, I think. . Now, of course, if you're creating a game, it makes a lot of sense to come up with a whole world of sound, like a whole sound world that fits your game world. But we don't want every app to sound like a game.

Now if you do add UI sounds, also add the option for people to turn them off. Now the last thing we want is for people to always hit the ringer switch, or even worst, delete your app because they don't like the way it sounds. So be very careful when adding these sounds. And if you do, add the option for people to turn them off.

Just as much as you pay attention to every single detail of the visual design of your app, do the same thing for sound.

Here's some things to keep in mind, especially when designing for a device like the iPhone. If I'm listening to music, and your notification sound comes in, the music ducks, which is fine because I choose to get these notifications. They give me information.

But if your notification sound is too long, it ducks for a very long time, so it becomes annoying.

So ask yourself the question, does it get in the way of the user in any way? We all know how frustrating it is if you're watching a video, and what you hear doesn't line up with what you see. .

You know, it gives us this uneasy feeling that something is not right here, and it's so much more satisfying when everything lines up nicely. .

So if you're adding sound to haptics or animation, then really try to find the right moment to synchronize to. We found that adding sounds to haptics, that pushing it back or forward by as little as ten milliseconds sometimes makes a difference in the experience, so really experiment with this a little bit and try to find the right timing.

Now for a couple more pro tips, this is a spectrogram.

It's a visual representation of a sound and of the frequency present in a sound.

The brighter the color, the louder that frequency. This particular sound sounds like this.

There's something that can be improved upon here. First off, there's a bunch of silence in the beginning. The actual sound starts here. And then I don't know if you heard it, but there was a lot of noise in the background. You can see it on the screen as well. It's the visual noise.

And then the sound was truncated at the end, so it cuts off. It sounds very unnatural.

Now a professional with some expensive software can easily make it sound like this. It's much cleaner. The background is darker, less noise there, and it fades out nicely.

But like I said, you need some expensive software for it, or do you really? Because we found that using Garage Band, you can actually come pretty close. You can adjust the beginning of the region, stripping out the silence, and then you can fade out the sound using track automation.

We found a preset called iChat Voice that has a denoiser function.

It's meant to be used for speech, but you can use it for any kind of sound, and we can come up with this. It's much cleaner. So if you're inventive, you can come a long way with a simple tool like Garage Band as well. Also pay attention to the frequencies you're playing. The iPhone speaker is a wonderful speaker, but there are some physical limitations, especially low frequencies. It may not translate that well, so it makes sense to filter out those frequencies that you don't use. This will clean up the sound and makes it sound more clear.

How do you find out what works and what doesn't work? Just test it out on the device that it's going to be played upon.

So if you're creating a game that will run on iPhone as well as on iPad, then test out the sounds on the iPhone as well as on the iPad, and realize that people will be wearing headphones, so also test it out for headphones.

Now when you're done exploring and done prototyping, maybe it's time to work with an expert. It's time to get serious.

So a sound designer can help you come with an overall sound concept and help you design the sounds, and a sound engineer can help you clean these sounds up and make them ready for everyday use.

Now we all know this guy, and even I have to admit I use it now and then.

Our goal is not for everyone to leave their sound on all the time, everywhere they go.

But when we start hearing more and more wonderful notifications and UI sounds that help our users and delight our users, then maybe, just maybe, you'll start seeing people reach for it a little less. That will be awesome. Thanks for listening to me today. [ Applause ]