Explore Nearby Interaction with third-party accessories

♪ ♪ Hi, my name is Yagil Burowski, and I'm an engineer on the Location Technologies team at Apple. In this talk, I'll cover the latest advancements, as well as the new features that are coming to the Nearby Interaction framework this year. The Nearby Interaction framework makes it easy to take advantage of the unique capabilities of U1, Apple's chip for Ultra-Wideband technology, and enables creating precise and spatially-aware interactions between nearby devices. It's the same technology that powers "Precision Finding" with AirTag and the fluid handoff gestures between iPhone and HomePod mini. This is what we'll cover in this session. We'll start with updates to the user permission flow. After that, we'll introduce new APIs for running Nearby Interaction with compatible third-party hardware.

And we'll finish with talking about resources for app developers, as well as accessory manufacturers, for getting started with adopting these new APIs in apps and third-party accessories. Before we dive in, for those of you who are new to Nearby Interaction, or just need a refresher, let's quickly recap how you use the framework in your app. You start with creating a session instance. That's the main object through which you'll configure and run your spatial interactions with nearby devices. Next, you'll need to designate a part of your app to conform to the NISessionDelegate protocol. The delegate will receive updates from the framework.

The next thing you need to do is create a configuration object, which will be a subclass of NIConfiguration. For example, you can create a NearbyPeerConfiguration when you want start a session between two iPhones that are running your app. Once you call run on the session with the configuration you had created, Nearby Interaction will start providing your app a stream of NearbyObject updates, each containing distance and, optionally, direction, to nearby devices that are actively participating in the session.

If you're interested in a deeper dive into the framework's APIs, please watch last year's WWDC talk, "Meet Nearby Interaction." To access data such as distance or direction to a nearby device, your app needs to get the user's permission.

So, let's talk about the improvements to the user-permission flow we've made this year. This is the Nearby Interaction permission flow in iOS 14. The prompt will appear the first time your app runs a session in a new app lifetime. The options on the alert, "Don't Allow" or "Allow Once," provide a very clear choice, but since this permission is one-time, it can also lead to additional prompts in certain situations. So, this year, there's a new way to grant apps permission to use Nearby Interaction. This is the new "Nearby Interaction" permission prompt in iOS 15. It looks kind of similar, but it's different in a couple of important ways. The system will automatically show the permission prompt on the first time your app runs an NISession. So, it's really important for you to make sure that the timing of when you run your NISession coincides with a clear user intent, so it's easy for your users to understand why a particular feature in your app requires access to Nearby Interaction. Let's take a closer look at the new options on the prompt. The new "OK" option grants your app permission while the app is in use. Whether the user accepts or denies your app's request to use Nearby Interaction, the permission prompt will not show again.

Starting in iOS 15, apps that use Nearby Interaction will appear in Settings. So, if users ever change their minds, they can head over to the Settings app and change your app's "Nearby Interaction" access. Make sure to test for this scenario when developing your app.

Let's summarize the new Nearby Interaction user permission model.

The new permission prompt, when accepted, gives your app persistent permission to use Nearby Interaction while the app is in use.

The alert will present a usage description string, provided by you in your app's Info.plist. In this purpose string, be clear and concise, and explain what are the awesome features in your app that need access to Nearby Interaction.

After the prompt shows for the first and final time, your app's name and icon will appear in the Settings app, which means that users can change your app's permission state at any time. When your app does not have sufficient permission to use Nearby Interaction, your NISessions will be invalidated with a permission-related error code. So, if a key functionality in your app relies on access to Nearby Interaction, be sure to explain this clearly to your users and guide them to the Settings app when appropriate. That was the new permission prompt in iOS 15.

And now, we're excited to introduce some new APIs that make it possible to use Nearby Interaction with compatible third-party hardware. We wanted this API to work with a wide range of accessories you may consider building. So, we based Nearby Interaction's third-party hardware support on standards we've been working on with industry groups.

And to help you get started on prototyping, experimenting, and building your accessories, we've worked with the chipset manufacturers, who are making available development kits containing hardware and firmware that is able to interoperate with U1 in iPhone. So, alongside development kits, we're also making available sample app code that works out-of-the-box with the hardware in the development kits. So, together, they can serve as an excellent starting point for your project.

And if you're an accessory manufacturer, please check out the Nearby Interaction, Accessory Protocol Specification document that will be available to you in "Developer Preview" on the Apple developer website. Whether you just want to tinker with this exciting new technology, or you're ready to integrate Ultra Wideband into your very own accessory, let's look at a simple example that illustrates how you use the Nearby Interaction API with a third-party accessory. Enabling different functionality based on the user's precise distance to an accessory can be a powerful way to streamline interactions with that device. So, let's say that you're building an accessory, and you want to integrate Ultra Wideband into it, to enable a set of features that will really make it stand out. And since Ultra Wideband can give you precise distance, you're planning to utilize it to keep track of a couple of precise zones or regions around the accessory. So, let's say that you define a region around the accessory that has a 1.5-meter radius, and another, larger one with a 3-meter radius. And let's say that you care about these specific distances because, when the user enters the larger region around the accessory, you want to enable some functionality A. And when the user enters the smaller region, you want to enable functionality B.

So, how do you actually achieve this in your app using the Nearby Interaction framework? First of all, Nearby Interaction expects your app and your accessory to have some sort of capability to exchange data between them.

As for the particular technology to use to exchange data, that is entirely up to you and what your accessory does and does not need to do. Suppose your accessory already supports Bluetooth. This will put you in a great spot because you'll be able to utilize your existing Bluetooth capability for your data exchange requirement. If your accessory is connected to the local network or is securely connected to the internet, you're in great shape, too. That capability to send data back and forth between the app and the accessory will serve you for what you'll need to do next.

Recall that when we wanted to start a session between two iPhones running your app, we created a NearbyPeerConfiguration. To start a session with an accessory, we'll create a NearbyAccessoryConfiguration.

This is a new NIConfiguration type in iOS 15. To instantiate an Accessory Configuration, we need to provide it some configuration data, describing the accessory that the framework expects to receive in a particular format called Accessory Configuration Data. But how do we get this configuration data, and what is this particular format? U1-compatible Ultra-Wideband hardware, like the types you'll be able to get from certified technology providers, will know how to generate this configuration data upon request. This means that code you're running on the accessory itself will need to generate this data and then send it over to your app on your data channel. Let's look at a code example for instantiating a NearbyAccessoryConfiguration with data received from an accessory. Let's assume that setupAccessory is a helper method I wrote in my app. Whenever I get the Accessory Configuration Data from an accessory, there's logic in my app that routes it to this function, along with the accessory's name. At this point, I can create an NINearbyAccessoryConfiguration using the data I received from the accessory. You'll notice that I create my configuration inside a do/catch statement. I do this because if the data I received from the accessory is invalid in some way, the NIConfiguration init method will throw an exception. But if the configuration object is created successfully, it means that the data I received from the accessory is in the right format and can now be used. The ultimate reason for creating a configuration is for using it to run a session. But before I do that, there's a good opportunity for me to cache some useful data for later use. The new NearbyAccessoryConfiguration type has a DiscoveryToken property for the accessory that the framework automatically populates. In addition, in the setupAccessory function I defined, I also have the accessory's name. So, it could be really useful to save the discovery token alongside the accessory's name at this stage because, later on, when I start receiving NearbyObject updates about this accessory, I would be able to use the token and the name to correlate the updates back to this accessory, and then display more relevant and rich UI.

Okay. So, we have valid configuration data from the accessory, and we used it to create an accessory configuration object. Now, we can get ready to interact with this accessory.

To manage the interaction, we'll need an NISession instance, and we'll need to set the session's delegate. To start the session, we simply call the "run" function on the session, using the accessory configuration object we created.

Just like Nearby Interaction needed configuration data from the accessory, the accessory also needs configuration data from Nearby Interaction in order to know how to configure itself. This data needs to be in a format called "Shareable Configuration Data." When you run a session with an accessory configuration, Nearby Interaction will provide the Shareable Configuration Data to your app through a delegate callback. And just like we used the data channel to receive the accessory's configuration data, here, we'll use it again to send the shareable configuration data back to the accessory. To make things more concrete, let's look at a code example. Shareable Configuration Data is provided to your app through the didGenerate delegate callback, which is a new delegate callback in iOS 15.

Here is a possible implementation of this callback in an app. The framework provides the Shareable Configuration Data, and also indicates which accessory it should go to, which is really useful in case you're interacting with multiple accessories. You should plan to send the data to the accessory as soon as possible over your data channel. On a general note, managing data connections to different accessories can take on many different forms, all depending on your use case. For simplicity, let's say that in my app's architecture, I chose to keep an independent data connection for each accessory I'm interacting with. And to keep my code nicely organized, I defined a helper function that gives me back a connection based on the NearbyObject I give it. Once I get a reference to the connection, I'll use it to send the Shareable Configuration Data to the accessory right away. It's important to optimize your app for sending the shareable configuration data to the accessory with as little delay as possible. If the ShareableConfigurationData is not sent quickly enough, your session may timeout. A timeout in a session with an accessory will be communicated to your app through the didRemove delegate callback. Let's look at how you might handle this scenario.

When Nearby Interaction gives me the didRemove callback, shown on the screen, I'll first check the "reason" parameter. If the reason is .timeout, and I have high confidence that the accessory may still be nearby, I can attempt to retry to interact with it.

To decide whether or not I should go into a "retry flow" with this accessory, I defined a helper function that contains specialized logic that helps me make this decision. You can imagine that conditions like "How many times have I retried without being successful?" or "Has the accessory notified me it has stopped?" or other similar questions can be a part of decisions like this.

If I do decide to retry, all I have to do is run the session again with the same configuration. Keep in mind, that the cached configuration will only remain valid as long as the session on the accessory was not terminated. If the session was terminated, I'll have to go through the same flow I used to create and run the session the first time. Keep in mind that sessions on the accessory is something the code running on the accessory has to manage, and can be done in many different ways, all depending on your use case.

Okay. You sent the Shareable Configuration Data to your accessory, and you even have logic to handle retries if needed. What's next? Just like with the Accessory Configuration Data U1-compatible hardware in the accessory will automatically know what to do with the Shareable Configuration Data once your accessory code provides it to it.

So, once the code running on the accessory receives the data, it needs to provide it as-is, and as quickly as possible, to the Ultra-Wideband hardware on board. You might be wondering-- how will the Ultra-Wideband hardware in the accessory know to generate or consume the configuration data? Both the Accessory Configuration Data and the Shareable Configuration Data are defined in a specification document we published earlier this spring. The document is intended for chipset and module manufactures, and it contains the necessary details for creating Ultra-Wideband solutions that use industry standards to interoperate with U1 in iPhone.

On top of the interoperability specification for chipset manufacturers, we are also releasing a specification targeted at accessory manufacturers. So, if you're hoping to build an accessory that contains Ultra-Wideband hardware, and you want to interact with it using the Nearby Interaction framework in your app, this document is for you. You'll find it on the developer website, alongside additional resources for getting started. This is where we left off. We had just sent the Shareable Configuration Data to the accessory over the data channel. Now, let's see what happens once the code on the accessory receives it, and provides it to the Ultra-Wideband hardware.

After receiving the shareable configuration data, the Ultra-Wideband hardware on the accessory will immediately start running in the appropriate configuration for interacting with the NISession in your app. And if both the accessory and the iPhone running your app are nearby one another, the session will start providing your app a stream of NearbyObject updates, containing distance and, optionally, direction to the accessory. You can even interact with several accessories simultaneously, by creating and running a session for each one. Depending on the hardware features on the accessory, you may also be able to get equivalent proximity updates in the code running on the accessory. Once you get NearbyObject updates from the framework, what do you do with them? As a reminder, we wanted to build an experience in which the app and the accessory enable functionality A when the user enters the larger region, and enable functionality B when the user enters the smaller region around the accessory. Let's look at a code example that shows how you could do this using NearbyObject updates in your iOS app.

When the session between the app and the accessory is running, updates about the accessory are delivered to the session's delegate through the didUpdate delegate callback. First, I'll grab a reference to the nearby object for which the framework is giving me an update. Next, I'll create a local variable with the distance to this object, which the framework is providing in meters. The next thing I'll do is provide this data to a helper function I have in my app I called getSmoothedDistance. I defined this function in my app to help me guard against rapid changes in the distance. For example, in cases where the user makes sudden movements, or they happen to be standing on the boundary between the zones. Lastly, I can check to see if the user's distance from the accessory crossed my predefined thresholds. Here, I can choose to enableFunctionalityA or FunctionalityB based on the current distance.

If you're interested in learning more about designing experiences based on distance and direction between devices, check out the "Design for Spatial Interaction" WWDC talk from this year.

So, let's summarize what we discussed in this session.

We talked about how the new permission model allows your app to access Nearby Interaction while the app is in use.

We took a deep dive into the brand-new world of Nearby Interaction-enabled third-party accessories, and what APIs you can use to build spatial interactions with your very own accessory. We also discussed third-party development kits, available from supported technology providers, and iOS app code that you can use to get started with your project right away. So, go on ahead, give it a try, and start experimenting and prototyping your use cases.

And when you're ready to adopt Ultra Wideband in your accessory, read the relevant specification documents, available on the Apple developer website. That's it for this year's Nearby Interaction updates. Thanks for tuning in. We hope this session was helpful and contained all the information you need to get started with building some amazing apps and accessories. [upbeat music]