Accessibility in SwiftUI

Good morning everyone, my name is Michael Gorbach and I'll be presenting today with my colleague John Nefulda from Apple's Accessibility team. I'm sure all of you are excited about SwiftUI and how it can help you build better apps faster and easier.

I am also really excited about SwiftUI and specifically, I'm excited about its potential for Accessibility.

We're going to tell you today how SwiftUI can help you build the next generation of apps for all of your users. We're going to kick it off with a quick introduction to Accessibility and our Accessibility technologies on our platforms.

Then talk a little bit about how SwiftUI make your -- makes your apps more accessible by default. John is going to come in and take us through the Accessibility API that we are introducing today. And, also take us through the Accessibility Tree that SwiftUI generates for us in our applications.

So, let's start with that introduction.

What do we mean when we throw around this word Accessibility today? Well, all of you have put a ton of work into your applications to make them fun and functional, helpful and a joy to use.

Given all of that work you want your apps to be useable by all of your potential customers. Not just some of them.

That's really what Accessibility is all about. It's about giving customers with disabilities full access to the value that your app provides.

And that means more happy customers for you.

It is also especially great for your customers with disabilities.

On our various devices like Macs and iPhones, Watches and HomePods, Apples technologies enable people with disabilities to live fuller lives.

They open up possibilities for them that did not exist just a few years ago. A huge part of how that happens in practice is through your applications. Yes, your apps can change people's lives. But for that to happen you need to do a little work to make them fully accessible.

When folks with disabilities interact with your applications, they will generally use Apple's built in Accessibility features. We've got a whole array of these shipping across our many platforms. I'm going to focus on one of them first called VoiceOver.

VoiceOver is a screen reader available across all our platforms. It's a product that lets folks with low to no vision use their Apple devices without needing to see their screens.

The way it does this is by reading to you. VoiceOver lets you step through the elements on your screen, taking each one of them in turn, and reads information to you about those elements like the titles of buttons, the names of images, and the values of text fields.

It also lets you use gestures on iOS or keyboard keys on the Mac to interact with those elements and kick off their associated actions. Here's VoiceOver on the Mac interacting with Calculator. VoiceOver is reading to us here and it's showing what it's reading in the bottom right as well. There are also two really cool new items in this long list this year. The first is Voice Control. I'm sure you all have seen the amazing demos.

The second -- yes, it is amazing.

The second is Full Keyboard Access. Totally new on iOS this year and greatly enhanced on the Mac. Full Keyboard Access lets you use your Mac of iOS devices entirely with just a keyboard.

It is a great, fast and fluid way to get things one on your Apple devices.

We are confident that this feature is really going to get used not just for accessibility purposes, but also just by folks looking to be extra productive on their Apple devices. Move around screens, kick off commands.

Here's Full Keyboard Access interacting with the Calculator on iOS. Notice the highlighted 8 key showing us where the keyboard focus is. These two great features are perfect examples of something we've learned on our team time and time again. Accessibility is for everyone. Not just for people with disabilities. That is why the little bit of work you do on your app for Accessibility has such a broad impact. Now, one of the best things about our platforms is that many of the features on this slide, including all the ones we just discussed, use the same underlying technologies. If you implement Accessibility in your app well it will work great with all of them. All these features ultimately rely on information that your app exposes to our platforms. The better this information, the better these features are going to work. We call this information your apps Accessibility User Interface. What actually goes into one of these? Well, let's take a quick look at Calculator just as a simple example, and take it apart.

The Accessibility User Interface your app exposes to us consists of Accessibility elements. Each view in Calculator forms one Accessibility element here. In this case, most of these Accessibility elements are buttons. So, let's take a look at one of them in particular. Here is the X button on the left and its Accessibility element on the right.

Notice that it's got a label for VoiceOver to read. You're going to want a label like this for all of your elements so a VoiceOver knows where they are in your app. In this case, the developers of the Calculator app have customized the default label from the simple X to the text multiply.

This element also has a type. On iOS that's described using traits and on macOS it's described using roles. That type here is button. And finally, this button element has a default action. Press on macOS or tap on iOS. A VoiceOver user can use this action to interact with the element. Let's take a look at one more. This is the display at the top showing us the results of our calculation.

We can see that the developers have given it a label named display.

It is also exposing another important piece of information called a value, that holds the current text its showing.

It has no default action since you can't right now interact with this element.

Default actions though are not the only way your Accessibility customers are going to interact with your app and its elements. You can add custom named actions to any Accessibility element in your app. For example, we could add a Custom Action on this one called clear, which would clear the display. So that is Calculator's Accessibility User Interface. Your app has one too. A lot of it is going to get created for you automatically, but with a little bit of work you can make it a whole lot better. You relentlessly polish the visual user interfaces on your apps. In a similar way you should polish your Accessibility User Interface as well.

Here are three key things to look for which we're going to show you how to do today.

First, your Accessibility elements should be understandable. They should have reasonable labels, values, and other describing information. An example of this is how Calculator changed the label of the X button to multiple so it's easier to understand. Your elements should also be interactable. That means, appropriate default actions and Custom Actions wherever they are needed. Adding that clear action to the main display as we just talked about is an example of this.

And finally, your elements should be navigable. Carefully thinking through the ordering of your Accessibility elements and grouping them as needed will really make your app more useable with our Accessibility features. So, if you're interested in how to do all this fun stuff with AppKit or UIKit we have some great sessions for you, but not this one. Today we are going to talk about SwiftUI. At Apple we have always deeply believed in the importance and power of Accessibility. That is why we all work extremely hard to make frameworks that let you folks make accessible apps easily and even automatically wherever possible.

Today, because of the high level of declarative nature of SwiftUI and the way that it understands your state, we can take automatic Accessibility to a whole new level with this new framework.

SwiftUI can make your apps more accessible by default and it also gives us a powerful foundation to do more of your Accessibility work for you going forward.

SwiftUI is going to generate your Accessibility elements for you, just like it generates NSView and UIViews for you. Let's take a look at this simple SwiftUI code.

It's creating a vertical stack, in that stack is a button, and it is sandwiched between some text.

SwiftUI is going to create three Accessibility elements for you here. One for each of the two texts and of course one for the button.

Keep in mind, the Accessibility elements that SwiftUI creates here are not UIViews or NSViews. Instead, they are SwiftUI's own Accessibility elements. We have all worked hard to make sure that SwiftUI's standard controls are as Accessible as possible by default.

Notice that although these elements are simple, they already have all the important information we need. They're understandable because they have labels.

They are intractable, the button has an action. And they're navigable because they're in the correct order. As I mentioned though, SwiftUI does have an Accessibility API to customize the elements it generates for you. Before we get into that though, I wanted to talk to you about the great Accessibility you get with SwiftUI before you write any Accessibility specific code. One thing you get for free in SwiftUI is notifications to help keep our Accessibility features like VoiceOver up to date regarding what is going on in your app. Remember, the Accessibility features I talked about earlier are all based on the Accessibility elements that your app exposes.

While running, VoiceOver's going to ask your app for that list of Accessibility elements. For example, when the user opens Calculator it'll ask Calculator for the elements on screen and Calculator's going to respond with those buttons we talked about earlier.

It's also going to send the element for the main display. It'll send a value with that. In this case, that value is 5. So, what happens if this information changes? For example, what happens when Calculator has gotten a press of the equals button and has updated the value of that main display? Let's say that value is now 10.

Well, VoiceOver needs to know that the value of that element has changed. That's why our frameworks support Accessibility notifications. Your app can tell VoiceOver, hey, the value here has changed.

We have been talking to you all conference about how SwiftUI is state driven. Well, this is where there are some huge benefits for Accessibility. Because SwiftUI really understands your state and is tracking when it changes, you never need to worry about Accessibility notifications again. Not even for custom controls.

Yes, exactly.

SwiftUI will track the state that you're using for your Accessibility and send them for you. Let's take a look at how this happens.

We have a toggle here and a button that are linked to a single piece of state. It's a Boolean and we've called it Enabled. You can see there are two Accessibility elements created for us here. In this case, we'll have an Accessibility element for that toggle. It's got a correct label and type. Its value is going to start off as 0. When you click the button the value of that Boolean enabled state is going to change. It'll go from 0 to 1 or 1 to 0. That's why we've called that button Flip. So, what happens here with Accessibility? Well, the Accessibility value of that toggle has now changed. So, we need to let VoiceOver know. Because it understands that that value's changed, SwiftUI will automatically send a notification to VoiceOver telling it exactly what happened. Notice, there's no Accessibility specific code needed here. You don't need to worry about Accessibility notifications in SwiftUI, because we send them for you.

Another area SwiftUI really improves on is Custom Controls. Often when you're designing your app, you may want to customize the look and feel of your UI. Maybe you want your app to have a unique visual style or you need to fit with a company theme.

If you have done Accessibility work in the past you know that oftentimes the more custom your controls in this way, the more Accessibility work you have to put into them. Well, SwiftUI's control styles let you have complete control over the look and feel of your UI while still getting great accessibility out of the box. That means you can finally easily make custom controls that are beautiful and unique, but still immediately accessible just like the system ones. Here's an example of a custom button style built using SwiftUI.

This API is giving us the style of the button including its label and whether it is pressed. Given that, we need to let the style know how we want this button to draw.

So, let's go ahead and put that label within a rounded rectangle and set some background and foreground colors that are going to depend on whether we're pressed.

We're also going to add a tiny bit of padding and set a nice big font and color as well.

So, you can see the result here on the right. Now that we've defined this beautiful custom drawing, we of course need to use it.

All we need to do is create a normal button and then use the button style modifier function to set the way we want the button to draw.

This is going to create an Accessibility element for us, which you can see in the top right here. Notice that despite the totally custom drawing here this Accessibility element is good to go. It has a label, it's got the right type, and it has an action.

Once again, we're doing this without writing any Accessibility specific code.

SwiftUI also provides better and more intuitive Accessibility for your images. Specifically, I want to talk to you about illustrations and glyphs which often form important parts of our UI's.

Imagine you have an app with a sign-up flow that ends in a green checkmark like this. Here are some simple SwiftUI code you might start with to create such a view. In this case, it's just a vertical stack with an image in it. After completing this sign up a sighted customer would look at this and say, oh sweet, that means I'm done. Great. But, a customer with VoiceOver will hear, CheckmarkGlyph.

That is not super useful. That is your developer created name for the image. I don't know what a glyph is or what it's supposed to mean. As a VoiceOver user I'm confused. Am I finished or not? Was there an error? This is not understandable for me.

The reason for that is that this image is not explicitly labeled. Here's the Accessibility element that's created by SwiftUI, which you can see it's got the right type but no label.

What SwiftUI is going to have to do here is pull the label from the filename of the image or the name of the asset.

That isn't really something we want to rely on. Because that name is for you the developer, not your customers.

With SwiftUI you can label your images now with localizable accessibility information right as you create them, like so. All we need to do is use the Label Based Initializer to image.

That way there's no need to remember to do this afterwards in some separate code.

This is going to create an Accessibility User Interface that is understandable.

Notice that this updated Accessibility element is now properly labeled. When I land on this image VoiceOver reads, sign up complete. On the other hand, sometimes your VoiceOver users don't need to land on an image. For example, maybe this checkmark is really just decoration and you've already got text below it that says, sign up complete.

The SwiftUI code might look something like this. A vertical stack with an image and two lines of text.

In this case, the image is really just decorative. You could create it like so, telling SwiftUI that with the right initializer.

Now, VoiceOver users aren't going to land on this image. If we look at the Accessibility elements, we've got two for the text, but none for the image.

This is now more navigable because your VoiceOver users aren't going to land on useless elements.

There is one last area where SwiftUI can make your life easier with Accessibility. And that is labels. Let's take a look at this control which is actually in our System Preferences. Its purpose it to let you pick a voice. You can see the Accessibility element here in the top right. It has no label and a value of Alex.

So, this is not understandable to me as a VoiceOver user. Because VoiceOver is going to read, Alex, and nothing else. Who is Alex and why do I care about him? If I'm using VoiceOver I don't know. I would have to scan around the rest of this screen to try to find some context. Let's take a look at the SwiftUI code instead for such a control. Here, we are using a standard SwiftUI picker which is being rendered as a popup button.

Now, SwiftUI's picker has a built in label we can use. In this case, we've set System Voice. Many of SwiftUI's controls have labels like this and the rest will soon. Let's take a look at the appropriate Accessibility element. This time it's got a label. And if we look at what VoiceOver reads; we have Alex System Voice pop up button. In an upcoming seed of SwiftUI you'll be able to do even more here. What if you want one single label to show for all of your customers, regardless of whether they're using Accessibility? Well, you will be able to configure that label to show visual -- let's say on the left here, and when you do so we will set up Accessibility for you. Specifically, we will set up a connection between the label on the left and the popup button on the right. They'll each be their own Accessibility element, but VoiceOver will know that they're linked. And so, VoiceOver's going to read, Alex System Voice pop up button.

So, here is what SwiftUI can do for you automatically. There are two things to highlight about all of these items. First, they're all consistent across SwiftUI's platforms. And second, to get all of this you don't need to write any Accessibility specific code.

Sometimes though, no matter how good our automatic Accessibility might be, it might not be enough. That's why we provide an Accessibility API for you to use. I'm going to hand the show over to John now to take you through it. John. Thanks Michael.

So, we've shown you all of the great built-in Accessibility that SwiftUI provides. It was designed to give everyone a good experience out of the box. But we understand that this alone isn't enough to be a complete experience.

This SwiftUI Accessibility API allows you to modify and ultimately create an even better experience for everyone.

So, I want to start off by showing a couple simple use cases for the API. And let's go back to the Calculator example that Michael showed earlier. We'll start with the multiply button, which by default will read X. To change the label, I'll use the Accessibility modifier function with the label parameter. And with this change the button will now read, multiply. We also want to provide VoiceOver for the selected state of the button. Visually the button becomes white when selected. We have an Accessibility trait that can help out with this.

And if you're not familiar, traits are a set of things that describe what your element is and what it can do.

Buttons for example, will automatically have the isButton trait by default.

VoiceOver will read, button, after reading out the elements label. For this example, we can add the isSelected state using the Accessibility function and add traits as the parameter.

VoiceOver will now read, selected, when the button is in this state. So, let's see how we can improve the Accessibility of the result view. We'll change the label to result and add an Accessibility value. You can use Accessibility value to provide VoiceOver with a readable description. You might choose to include units or other additional context.

But for our purposes, the number itself is sufficient.

Finally, we'll add a Custom Action using the Accessibility action function.

Using the name parameter, we'll create a Custom Action called Clear.

VoiceOver will read that there are actions available when focused on this element.

And this was just a quick overview of how to add Accessibility in SwiftUI. The Accessibility function adds information and the Accessibility action function allows you to add actions. But I want to talk a little bit more about when to use this API and how you can identify the areas of need within your app. And this comes back to what your customers need your apps to be. Understandable, interactable, and navigable.

And here are a few questions to help you evaluate this.

Do the displayed strings provide enough information? Will a Custom Action simplify the interaction? And can you speed up navigation? So now I'll show you a video of VoiceOver moving through an app that I created. And as it does, try to keep these three questions in mind.

So, this is an app that calculates the contrast ratio between text and its background color. At the top you can see the current value of the ratio and at the bottom there are two areas where you can adjust the red, green, and blue values of the background color and the text color.

I've also added a double tap gesture on the top view to swap the two colors.

So, let's see how VoiceOver uses that by default.

:50 11.7 1. 11.7 1. 11.7 1.

Reset colors button.

Background. Red 76. 30 percent. Adjustable. Green 217. 85 percent. Adjustable. Blue 100. 39 percent. Adjustable. 39 -- 100 percent.

So, there were a couple of things in the video that stood out to me with the label at the top. First, there was a symbol that wasn't read out. VoiceOver doesn't always read out your punctuation depending on settings. And in a situation like this, where the punctuation represents that this is a ratio, we should pass that information on to VoiceOver.

So, 11.7 1 should read 11.7 to 1.

And secondly, the numbers were spoken without any context.

It's probably a good idea to give this a proper label to describe its value. So, we can use contrast ratio.

And we'll do this by calling the Accessibility function twice on a contrast ratio view. Once for the label and another to add the value. Another thing that stood out to me was the information provided by the slider.

The value of the slider was a percentage, but in this case, it actually represents the color components value from 0 to 255. We can also hide the label above the slider and expose that information through the slider instead. So that'll group all the information into one element.

So first let's hide the label.

We'll call the Accessibility function with visibility as the parameter.

Passing in .hidden will hide the element and any children that it may have. Next, we'll add the information of the label onto the slider.

We'll give it the label of the color that it will modify and a proper value between 0 and 255.

We also have a double tap gesture on the view. Gestures on iOS can be challenging to perform for customers who use our Accessibility products.

Some of our products have a built-in gestures menu to help perform some of the more complex gestures.

But we can do better here. An easy simplification for this is to create a Custom Action for the gesture. This helps in two ways. First, the name of the action will help describe what will happen when you activate it. And second, actions are available everywhere in iOS and they're discoverable in all of our Accessibility products.

So, let's add the swap colors action to our contrast ratio view by calling the same Accessibility action modifier function. And finally, can we speed up navigation here? Looking at the view it can be divided into three main spaces; the contrast ratio, the background color, and the text color.

So, these are good candidates for the isHeader Accessibility trait. This trait's used by VoiceOver to navigate quickly between different header elements in your app.

So, let's add the trait to the contrast ratio view and the two text views in each of the color picker sections.

So, let's take a look at the results of the changes that we made, starting off with the top view.

Contrast ratio 11.7 to 1. Heading.

So, the element now provides context and a proper description for the value. Let's move onto the sliders and we'll adjust the blue sliders value.

Reset colors button.

Background. Heading. Red 76. Adjustable. Swipe or down with one finger to adjust the value.

Green 217. Adjustable. Blue 10. Adjustable. Swipe -- 100. 255.

And finally, let's use the VoiceOver rotor to navigate by headings and swap the colors using the Custom Action that we created.

Adjust value. Headings. Text. Heading. Background. Heading. Contrast ratio 13.0 to 1. Heading. Actions. Swap colors.

So, with just a few tweaks we were able to greatly improve the Accessibility experience of this app. And these are just the basics of the API. But I want to move onto something a little bit more conceptual. The Accessibility Tree.

And you've seen how Accessibility elements are created from your views. But I want to go in a bit further and talk about this tree structure of these elements and how you can manipulate the structure to provide a better experience. So, we'll start with how this tree's created. Starts off with your view tree created directly from your code.

On the left here is SwiftUI code for a table cell view.

It has one label, a spacer, and two buttons.

Arranged horizontally inside of an HStack.

And this directly translates to the view tree structure on the right where the HStack is the parent node and the label, spacer, and buttons are its children.

Because of our built in Accessibility, this view tree will create three Accessibility elements. One from the text and two from the buttons.

These elements will become children of your platforms hosting view and that's passed on to our Accessibility products.

This subtree may also be a part of a larger Accessibility Tree, mixed in with UiKit or AppKit elements. The SwiftUI tree can even be a parent of other platform elements bridged over using a UI or NSViewRepresentable. And you'll notice the levels of hierarchy in this tree and you might be thinking about how quickly this can become complex. But I'll show you how creating hierarchy in this tree can actually improve navigation.

So, let's take a look at our example again.

The table cell itself has three Accessibility elements. VoiceOver will navigate these in order and this works great if you have only one cell. But it's more likely that you'll have hundreds of these in your app.

And as you can see, there are a ton of elements in this tree. And it doesn't seem very useful to have a lot of repetitive information with the follow and share buttons. If you landed on this follow button, how would you know if it was for person 5 or person 6? There's no context here unless you look around.

But we can make this a lot simpler. And we'll do this by creating an Accessibility element at a higher level than in the view tree.

We can find a view that is a parent of our Accessibility elements and create a new element there.

In our case, we'll use the HStack. We can use the Accessibility element function on our HStack to create a new element.

Specifying .combine will merge the Accessibility provided by the children into one element.

In this case, the text view will provide the elements label and the two buttons will automatically be converted into Custom Actions.

And now, VoiceOver will see just one element here which will read the person's name and that there are actions available.

And with these changes we've significantly reduced the number of elements in this tree. You can navigate the cells one by one as expected.

You can access the buttons through Custom Actions. So, grouping related UI like this is a simple yet great way to improve navigation in your app. Another important component of navigation and Accessibility is ordering.

There may be situations where you find that the default ordering isn't giving you the experience that you'd expect.

So, you may have noticed from the video earlier that the reset colors button was actually the second element in the swipe order.

The two elements are arranged in a ZStack which orders elements from back to front by default.

Visually though, the button appears above the contrast ratio. So, in this case we might want to swap that.

To customize this order, we can use sortPriority.

So, all elements have a default sortPriority of 0. And elements in a container are sorted from highest to lowest priority.

So, we can set the sortPriority by calling the Accessibility modifier function with the sortPriority parameter.

So, we'll give our button a sortPriority of 1 and since the contrast ratio has a priority of zero, the button will be scanned first and the ratio will be scanned second.

So, the Accessibility API helps you make your apps more understandable.

It lets you add context by adding labels, values, and hints using the Accessibility function. You can use the Accessibility action function to add actions and make your apps more interactable. And finally, you can use the Accessibility element function to group your elements together and improve navigation. With built in Accessibility and simple enhancements with the Accessibility API, SwiftUI lets you make your apps accessible, faster and easier than ever.

Now, back to Michael.

All right, so let us close things up here. One last thing before we do that.

If you have put in all of this work into the Accessibility of your app, how do you know you're actually being successful? Well, we have two pieces of advice for you here. First, there is no substitute for using your app. That means trying it out with our real Accessibility features like VoiceOver, Full Keyboard Access, and Voice Control. For example, if you're using your app with VoiceOver and there's something you can't do with just your voice or you -- there's something that you need to look at the screen for, you have some work to do. Or if you're using your app with Voice Control and there is something you can't do with your voice. Or you're using your app with Full Keyboard Access and there's something you can't do with your keyboard; you have some work left to do. Second, there is the Accessibility Inspector. This is a product that ships with Xcode and can help you dig deep into the Accessibility of your app. The Accessibility Inspector doesn't replace testing your app with our real Accessibility features, but it can be a powerful tool to understand your Accessibility Tree and the debug when things go wrong. There are some great sessions about the inspector on this slide. So, a few points to summarize what we've talked about today.

We have worked super hard at Apple to build SwiftUI so it can make your apps as accessible as possible by default.

And we've designed SwiftUI with Accessibility in mind from the very beginning. We've talked today about what a great Accessibility User Interface is. I know we've been drilling it into you, but remember, understandable, interactable, and navigable.

To go beyond the basics and add polish, SwiftUI has a powerful Accessibility API for you to use.

And that API is now finally unified across all our platforms. So, you only have to learn it once.

Two useful references going forward. First, we will be releasing sample code for the session. You can find it at the link above. And second, we have an Accessibility lab coming up immediately after this session. Come join us to meet us, talk about SwiftUI, and Accessibility. We are looking forward to meeting all of you. We're super excited of the new era of accessible apps that you folks will create with SwiftUI. Thank you for listening.

[ Applause ]