공간적 상호작용을 위한 디자인

♪ Bass music playing ♪ ♪ Peter Tsoi: Hello and welcome to "Design for spatial interaction." My name is Peter, and I'm a designer on the Apple Design team.

Together with my colleagues Arian, Taylor, Linus, and Pedro, we're excited to share some of the considerations that guide how we approach designing spatial interactions.

We'll share lessons learned while creating interactions for AirTag, HomePod mini, and iPhone, and also give you tips and tricks that will help you apply those principles while designing interactions of your own.

Let's get started.

As our devices have evolved, so too has the way that we interact with them.

Early computers were operated only using a keyboard and interfaces were entirely text based.

Users memorized abstract commands and relied on arrow keys to move around the screen and express their intent.

Nowadays, most people use their keyboards together with graphical user interfaces and a mouse.

These innovations removed a layer of abstraction and allowed people to use the mouse to point, click, and drag to more directly express their intent.

Later, Multi-Touch brought the digital world to people's fingertips, removing yet another layer of abstraction and allowing people to touch their music, their photos, and the mobile web.

Today we want to talk about a class of interactions that go beyond the bounds of a single device and ways to use the capabilities of the latest Apple devices to help users interact more directly with their physical surroundings.

Some of these interactions may already feel familiar, like how you can pay for a coffee by holding your phone or watch close to a payment terminal.

Or how you can quickly check the battery level on your AirPods just by flipping open the case when it's near your phone.

These human-scale interactions -- the ones that happen between devices nearby -- have the ability to remove layers of abstraction and allow users to interact more directly with their surroundings.

And with the enhanced spatial awareness enabled by the U1 chip in the latest Apple products, we've been able to deliver more capable and responsive experiences for AirTag, HomePod mini, and iPhone.

And now, thanks to new APIs and the ability to interact with third-party hardware that you build in iOS 15, you can now bring these kinds of experiences to your own products.

In designing these interactions, we learned that it's important to consider distance and ability, provide continuous feedback, and embrace the physical action.

Let's dive right in.

In iOS 13, we made it even easier to share content by bringing spatial awareness to the share sheet.

Using information about who you are facing or physically close to, combined with on-device knowledge about the frequency and recency of your communications, allows your device to intelligently predict who you are trying to share with.

This is a great example of how existing features in your apps can be made more intelligent with spatial awareness.

You don't have to build an entirely new experience to take advantage of these new capabilities.

When incorporating spatial awareness into existing features, be mindful that these capabilities are not available on all devices and your design should accommodate varying levels of capabilities.

With the share sheet, all devices are able to look for others using Bluetooth and Wi-Fi.

However, iPhones with the U1 spatial awareness chip take this a step further by prioritizing others that you are facing or are very close to.

Of course, spatial awareness can be used to not just enhance existing experiences; it can help build entirely new ones.

Next, Arian will share how these same concepts apply to AirTag.

Arian Behzadi: With AirTag, we will show you how your design can adapt to accommodate different distances and abilities.

In this example, imagine I've misplaced an item I care about.

I retrace my steps and realize I'm not sure where it is.

I use my iPhone and feel relieved that I can see where it is on a map.

Previously, I'd only be able to get to the general vicinity of this item, but now my iPhone can guide me at a scale that is much more helpful.

In fact, the same button that gave me directions when I was far away has now become a button that will help me find it as I'm closer.

Augmenting elements of the design when at the appropriate scale is a great way for people to discover and use these new abilities.

As iPhone is locating my AirTag, I'm invited to move around and search my surroundings.

Once connected, an arrow forms and points directly to where I need to go. As I align with the direction of my AirTag, the design lights up to reinforce this.

We emphasize this facing direction and provide guidance when not facing this direction.

In a subtle way, this distinction also scales based on distance.

When further away, we found that it is really hard to remain facing a specific direction.

To remedy this, we are actually more generous with the angle that iPhone considers facing.

As you get closer, this forgiveness becomes more narrow to become more and more specific.

This happens without you noticing as you're gently guided to your AirTag.

When you're finally within arm's reach of your AirTag, the arrow defers to a form of feedback that we find much more effective at this small scale. The design transforms to highlight haptic feedback as you pass your iPhone over where your AirTag might be hiding.

With this example, we've shown you how your design can transform based on distance, whether you're very far away or within millimeters.

Think about how your design can best accommodate varying abilities we all share as humans.

Be forgiving with angles at a distance, and consider changing the dominant form of feedback to one that works best at smaller scale.

Next, I'd like to hand it over to Taylor to show you just how critical this continuous feedback is when designing these kinds of interactions.

Taylor Carrigan: Thanks, Arian.

We've gone to great lengths to make onscreen interactions fluidly and dynamically respond to touch input, but this continuous feedback is even more critical when interacting in the physical world where your device becomes a more literal extension of your body.

The right type of feedback applied and choreographed at the right time throughout these interactions can help make the feature you're designing discoverable, provides instruction, and can communicate success or failure.

It also acknowledges that our movements can start, stop, and be interrupted at any time.

We're going to talk about how you can use feedback you can see, hear, and feel to help connect your interaction to the physical world.

Though dependent on the capabilities of the devices used, there are multiple types of feedback worth considering and using together.

These can include visual feedback -- such as user interface changes, lights and hardware interactions, and visual feedback coordinated across both devices -- audio feedback , and haptic feedback.

We'll also talk about natural interruption and cancellation and the importance these play in creating a successful interaction between two devices.

Let's look first at how different types of feedback help make the transfer of music to HomePod mini intuitive and satisfying.

With HomePod mini, we use multiple types of feedback to establish a physical relationship with your iPhone, provide direction, and ultimately confirm that your music has been successfully transferred.

We achieved this by creating two discrete boundaries around the HomePod, and respond to your movement across and between them.

Feedback is provided when your iPhone reaches the first zone, between the first and second to instruct you to move closer or allow you to cancel the interaction, and finally when your iPhone reaches the second zone, to confirm and transfer music.

Feedback is continuously provided from the moment iPhone reaches the first distance threshold.

On screen, the banner position, scale, and modality created by the background blur increase fluidly and in direct response to your distance from the HomePod.

When designing visual feedback, look for opportunities for the onscreen elements to animate in a way that feels related to your physical movements -- like how the movement of this banner is related to the linear movement of your hand towards or away from the HomePod.

Haptic feedback complements this with a physical acknowledgement that the two devices are aware of each other, and increases in strength to encourage you to continue moving closer.

In addition to the feedback on iPhone, HomePod mini acknowledges the proximity of iPhone through the animations on the top of the speaker.

Great spatial interactions use natural body movements to not only confirm but also to cancel actions.

When moving an iPhone near a HomePod mini, clear and continuous feedback that tracks your distance from the HomePod makes it clear that you can simply pull away to cancel or interrupt the gesture.

This allowed us to create an interaction that can be done entirely without looking at the display or requiring additional onscreen buttons to cancel or confirm user intent.

All of these aspects are driven by and respond to the movement of your body as it brings iPhone closer to or farther away from the HomePod.

The directness of the relationship between someone's movements in the physical world and what they're seeing, hearing, and feeling on one or both devices are critical when designing a spatial interaction.

Next, Linus will talk about the unique constraints of providing feedback while finding an AirTag.

Linus Persson: HomePod mini is a good example of how to choreograph feedback across two devices, both capable of expressing rich feedback.

For designs where this is not a possibility, you might have to rely more heavily on a single device.

Since AirTag helps you find an item when it's lost, the experience is designed for scenarios when it's hidden from view.

Let's revisit the finding experience we saw before, this time with an eye on how we provide continuous feedback across the senses using iPhone's rich haptic, visual, and acoustic capabilities.

A spatial interaction feels great when it's responsive and follows your motion.

That means responding to movements that are big as well as ones that are subtle.

Even as iPhone is connecting to the AirTag, the interface gently rotates and responds as I turn, implying the nature of the interaction that is about to follow.

These dots also provide the building blocks for a continuously adapting experience.

Once connected, the distance between me and my AirTag also appears.

This distance responds incredibly precisely as I walk in any direction.

It instantly updates and tells me if I'm walking toward, or away, from where I want to go.

As I walk around, the same dots smoothly form an arrow pointing me in the right direction.

This moment is reinforced further using a delicate haptic and sound to mark its importance.

It feels as though the arrow floats in space as my iPhone moves around it.

As you saw earlier, the interface reinforces moments when I am on the right path.

When facing the direction of my lost AirTag, the screen boldly lights up and I also feel and hear the arrow snap into place.

By responding to every step and subtle turn, the design feels tightly coupled with me and the space I'm in.

When designing your spatial interaction, consider how people will connect their own movement with what they experience on the device.

Try to make a clear link between action and feedback in a way that is mindful of the particular motion.

Just ahead of the arrow, there's a dot that starts to pulse in my hand.

The pace of this pulse picks up as I get closer, and the haptics become tighter and crisper along with it.

If I break away from this direction, the pulse stops and the interface guides me back.

If I walk far enough away from my AirTag, the arrow disassembles to mark the loss of signal.

Nuance in moments like these also provide important and very helpful guidance.

Be mindful of how feedback can complement human motions that are varied and unpredictable, and build a design that is resilient and can adaptively change to accommodate the way we move.

When within arm's reach, we visually zoom in on this dot, and the pulse changes to continuous haptic feedback that shifts with my movement.

This haptic response changes in character as I move closer and further away.

I can move my iPhone over where AirTag might be and sense through my hand more precisely where it's hiding.

Note that we do not use sound for this part of the experience, as haptic feedback does a much more effective job.

As with any continuous feedback, your design should be mindful of how haptics, sound, and visuals work together in concert across the senses.

Similarly to the way we perceive what we see, hear and feel in the world as one holistic experience.

When designing your interaction, it is important to consider the different strengths of each form of feedback and how they can be used together.

Seek to provide a visual layer of continuous feedback tied to physical motion and use additional feedback to emphasize important moments in your interaction.

Use haptics and sound judiciously and at the right levels, with a repeatable cause and effect so it is clear what is being communicated.

In addition to working with feedback in this way, keep in mind that your design will complement a physical action.

And to speak about this is my colleague Pedro.

Pedro Mari: In the case of HomePods, we use spatial awareness to make the task of selecting a speaker more natural.

Oftentimes, I'll be listening to some music, and I'll want to move it from my iPhone to a better speaker near me.

Previously, if I wanted to play this song on a different speaker, I'd have to go into the AirPlay list.

In this list, all of the other devices in my space are presented equally, whether they are near me or not.

What I am really interested in, though, is this particular HomePod.

With this new human-scale interaction, I can finally transfer a song to this HomePod without navigating an onscreen UI, or having to interact with the screen at all, for that matter.

We provide visual feedback on the HomePod itself to show that the music is about to be transferred.

The light modulates precisely with my movement, growing in intensity as I approach, and shrinking as I move away.

And of course, the ultimate confirmation of the transfer is the music playing from the speaker itself.

By embracing the physical action, a previously intangible experience now feels completely tangible, visceral, and instinctive.

Think of ways that your own interfaces can support a natural physical motion, rather than an abstracted list of options.

Try to make your experiences and designs defer to the physical task at hand.

Also, consider how the concepts of "this" and "that" can be finally used to create a more natural experience that aligns with how we think.

Make sure to provide instant and continuous feedback on both devices.

Specifically, be very clear on the target device that an action is happening, as this is where your attention will be naturally drawn to.

Similarly, when finding a lost item, my attention is meant to be directed at the world around me and where it might be hiding.

Playing sound from the AirTag gives me an immediate sense of its place in space, drawing my attention to my surroundings.

When it comes to the visual feedback on iPhone itself, make sure it can be read when performing a task that extends beyond the bounds of the display.

We often design our interfaces to work in this manner when we anticipate it will be viewed in the periphery.

For instance, when using turn-by-turn navigation, the typography of when and where I need to take my next turn is bigger and bolder than other notifications that might occupy the same space.

The buttons and active area of TV Remote are extra large and are designed for your eyes to be directed to the TV itself, rather than the control for it.

The digits and operation buttons in Calculator are large and bold, as we anticipate you might be referencing numbers on a restaurant bill to calculate a tip, for instance.

As in the previous examples, the type size we use for the distance when finding an AirTag is set much larger than in other parts of the system, and the entire UI revolves around the central element of a giant arrow.

Bold color changes can be read in my periphery.

Sound reinforces key moments and states -- I can actually feel the distance getting smaller as I walk without ever having to look at the screen.

This ultimately supports a more natural pose as I use it.

When in arm's reach, haptic feedback can guide me closer even if the screen is not in my periphery.

These large and bold UI choices enable those with varying abilities to access the experience as well.

Make sure your design uses more than one mode of feedback, and that each one is salient enough to be clearly understood.

Your design shouldn't demand too much attention to communicate its information or compete with the primary task.

Lastly, try to reinforce good states and celebrate positive progress along the way to completing a spatial interaction.

Always consider the physical action your experience is complementing.

And now, back to Peter! Peter: We've covered a lot today.

We've seen how new and existing experiences become more intelligent and relevant when they consider the device's context and surroundings.

You learned how responsive and continuous feedback can help users discover and use these new experiences.

And finally, we took a look at techniques that allow your design to be effectively experienced peripherally while keeping attention on the physical surroundings.

We hope that you are just as excited as we are about how spatial awareness can simplify and bring more directness to interactions between our devices and our users' physical surroundings.

For more information, be sure to visit developer.apple.com where you'll find the Human Interface Guidelines for spatial interactions, as well as technical information about how to implement these types of interactions.

We can't wait to see what you'll build next.

♪