Explore advanced App Intents features for Siri and Apple Intelligence

Hi, I'm Antonio Cancio, a software engineer on the App Intents team. Let's "Explore advanced App Intents features for Siri and Apple Intelligence." Today, you'll learn techniques to take the experience of your app in Siri and Apple Intelligence beyond the basics, so it feels polished, personal, and unmistakably yours. This talk assumes a basic understanding of App Intents and App Schemas. If they're new to you, I recommend starting with these sessions which cover the fundamentals.

Let's go through today's plan. First, we'll explore how you can make people's conversations with your app through Siri more intuitive and familiar. You'll learn how to build custom responses that match your app's look and feel, and how to set up interaction donations that help Apple Intelligence feel more personal.

Then, we'll discuss how you can make your content more widely available. From semantic index integration to structured search, in-app search, and onscreen awareness, you'll come away knowing how to help Siri find your content and connect what's visible on screen to actions it can understand.

Finally, people can use Siri from anywhere in the system.

We'll cover how to add entity annotations to existing integrations with notifications, Now Playing, and alarms, so people can act on your content wherever they encounter it. To show this in practice, I wanna share a few apps that my colleagues and I have been building: CosmoTunes, which lets me play music, and create alarms and timers with my favorite songs; UnicornChat, a messaging app to stay in touch with friends; and CometCal to manage my calendar. Hm. Hey, there's something about App Intents that encourages celestial symbolism. You can download these samples from the links below and follow along.

Let's start with shaping the Siri conversation using custom responses.

Siri does the heavy lifting. It understands natural language, picks the right action, and crafts a helpful response. The App Intents framework gives you the tools to shape what Siri does, and refine how it responds. By tailoring how it responds, you can let your app's unique personality shine through. Let me illustrate this with some code. In CosmoTunes, .addToPlaylistIntent lets people add songs to a playlist. To start, I want Siri to handle the response. In the perform method of the intent, I'll add the song to the playlist, and return an empty IntentResult.

This tells Siri to take care of the response when the intent runs.

After trying that out, I want the response to better match the app's personality. I call songs tracks and playlists mix tapes. To customize this, I mark the perform method as providing a dialog response by adding the ProvidesDialog protocol. I'll also change the IntentResult by passing an IntentDialog containing both full and supporting strings. Siri can show the supporting string with UI, and read the full dialog on voice-only devices like AirPods. Because of this, the full string should describe what happened on its own.

That covers responses when your intent has finished. But what if you want to ask people a question while your intent is running? A well-placed clarifying question lets people finish the action they meant to take.

To ask a question before your intent result, use a dialog request within your perform method.

People can create timers in CosmoTunes that start or stop audio playback after a set amount of time. This intent adopts a schema with required and optional parameters. If there's a timer already running, I want to ask the person to name this new timer to avoid confusion.

I'll request a value for the optional label parameter when one hasn't been provided. If you want to ask people to choose from a list of items, or ask for a confirmation, check out the sample app and documentation to learn about other kinds of dialog requests. Next, I want Siri's visuals to match the app's look and feel. Entity display representation and custom views in intent responses are a great opportunity to visually present information and surface your app's identity alongside the dialog.

Defining an entity's DisplayRepresentation can tell Siri how your entity should look and read when showing your content. Entity display representation can be used in responses, like when an entity has been created or updated. They are also used when asking someone to choose between similar entities, or when answering questions about content in your app. Spotlight and Shortcuts can use them, too.

To define a basic DisplayRepresentation, provide a title.

You can make these even richer by providing a subtitle and an image.

I'll provide an image associated with my songs that Siri can show when people ask about their songs in the app.

Entity DisplayRepresentation lets you refine the visual identity of your entities across the system. To define the visual responses of some specific actions, you can use custom view snippets.

Back in AddToPlaylistIntent, Siri already responds automatically using the entity display representation.

To use a custom view, I add the ShowsSnippetView return type to my perform method.

This lets me return an IntentResult with a SwiftUI view, like my PlaylistSnippetView which displays the playlist details in familiar colors. When approaching customization, test your intents and decide where customization actually makes sense for your app. Make sure your responses are accurate and sound natural across all platforms, including voice-only devices like AirPods. Remember to ask clarifying questions sparingly to avoid friction.

Finally, use custom visuals to bring your app's identity to Siri, keeping in mind how they'll scale across the ecosystem.

Your intent responses typically come at the end of a Siri interaction. But, Siri can ask additional questions before even calling your intent. For example, if I ask to message someone, Siri may ask me to choose from a list of similarly named contacts, because the system isn't sure which one I mean.

That brings us to Interaction Donations, an API you can adopt to help Apple Intelligence be even smarter in handling those kinds of requests.

Here's the good news. When people interact with your app through Siri or Shortcuts, the system already knows about it. But, Apple Intelligence can't learn from actions people take through your app's UI without your help. That's where donations come in.

Each donation is a hint that a person took a specific action in your app's UI. The system stores these as schema-conforming App Intents in a temporary transcript, giving Siri the context it needs to make smarter decisions. The transcript contains UI interaction donations over time. Say someone opens UnicornChat and sends a message from the compose view. Right then, the app donates the send message action to the system, using the SendMessageIntent schema.

After messaging them frequently in the app, eventually, when someone says: "Send a message to a contact from the Home Screen," Siri might infer the right app to use for that contact.

To adopt interaction donations for message sending, I started by looking at the ConversationView in UnicornChat.

The ConversationView and the sendMessage intent both call the same helper to send the message.

That looks like a good place to start donating the sendMessage intent.

I'll add a donateIntent parameter so I know whether the helper was called from the intent or from the UI. Apple Intelligence already learns from Siri interactions, so I only need to donate UI interactions.

Then, I'll create the intent, populate its parameters and the intent result, and donate it via the IntentDonationManager API. Now when a person opens the app and sends a message, the system can learn when they prefer to use UnicornChat.

Beyond learning preferences, Interaction Donations also keep Siri aware of ongoing activities in your app. This is especially useful for activities people might start or stop with Siri.

In an app in the Maps domain, a person could start a NavigationSession with the app's UI which donates that interaction. Then, the person gets in the car, and asks Siri to add a stop on their way. Thanks to the Interaction Donation, Siri can know what NavigationSession is active in the app, and help the person with their request.

This pattern applies to intents that start or stop NavigationSessions in the Maps domain, and stop, start, pause, or lap stopwatches in the Clock domain.

Your interaction donations should accurately represent real user behavior in your app. If your app donates excessively, the system may ignore those donations. Once Siri has gathered all the parameter values and is ready to call your App Intent, there is one final step: confirmation.

Asking people to confirm that the action looks right keeps them informed and protects them from unintended side effects, which are a known risk with Large Language Models.

This matters most for intents that could have meaningful side effects on your data, or the outside world.

That's why Siri can automatically confirm these kinds of intents. For example, it can confirm when I say: "Cancel my expedition next week in CometCal." This matters even more for intents that update app content which a person has made public or shared with others.

For example, Siri may not confirm when I update a personal event, but it may confirm when I ask it to update Crew Lunch since I'm updating an event with attendees. By default, Siri assumes your entities are private to the person, and may skip confirmations for them.

To tell Siri the owner has made an entity public or shared it with others, conform the relevant entities to the new OwnershipProvidingEntity protocol.

Only add the protocol to entities that people can share or make public in your app. Then, provide the ownership state.

Keep the ownership state up to date whenever the system requests an entity from your app.

This ensures Siri has the necessary information when deciding to confirm.

Remember those entity display representations we customized earlier? Siri can use them as visuals in these intent confirmations, too.

Giving people a chance to confirm actions when appropriate builds trust in your app's experience with Siri.

To learn more about other ways to establish trust and mitigate risks, check out "Secure your app: Mitigate risks to agentic features." So far, I've defined how the actions in these apps work with Siri, given Apple Intelligence context about how people use my apps, and helped Siri protect people from unintended side effects. Next, let's talk about how Siri finds content in the first place. There are three paths I want to cover: the semantic index, structured search, and in-app search.

Playlists in CosmoTunes are all available locally on device. To help Apple Intelligence find them, I'll adopt IndexedEntity and index those entities in Spotlight.

I'll use the .indexAppEntities method on CSSearchableIndex, which populates the Spotlight semantic index.

Now I can ask Siri: "Play my WWDC playlist in CosmoTunes." I can also search for my playlists in the Spotlight search UI. And depending on the App Intents domain, indexing entities in Spotlight provides semantic search capabilities. This means Apple Intelligence and Siri can understand your entities based on meaning, not just exact keywords.

Adding to the index is the first step. Keeping it up to date is key to helping Siri find your content.

Index new entities as people add content to your app. Update existing entries when key properties change, especially those used in your display representation.

When people remove content, delete those index entries too. Spotlight may need your app to reindex its entities. Your app can support reindexing by adopting the new IndexedEntityQuery.

Check out IndexedEntityQuery in the sample project.

If your project already supports reindexing with Core Spotlight-level APIs, you do not need to define an IndexedEntityQuery.

However, you might not index your entities if your content dataset is large, lives on a server, or changes too frequently to index ahead of time. For example, I decided to index all the app's playlists, but not songs. To still give people the flexibility in playing songs with Siri, I reached for an IntentValueQuery.

IntentValueQuery is suitable if you don't index all your entities ahead of time. This is very similar to EntityQuery. The key differences are that your app receives a structured search input from the system, and you can return more than one entity type. Siri needs an entity for the audioEntity parameter on the PlayAudioIntent in CosmoTunes.

To find the entity, Siri calls the IntentValueQuery with an AudioSearch. The query maps the structured properties of that search input to audio entities in the app.

In the IntentValueQuery, I implemented the values for method to handle the AudioSearch input, and return an AudioEntity.

AudioEntity is a UnionValue type that includes both songs and playlists. The AudioSearch value has a .criteria property that describes the person's query. The .searchQuery case contains the relevant part of what the person said, and I use that to find matching entities. The app also supports an unspecified search. For example: "Play CosmoTunes" which isn't specific about what I want to play. In that case, the app jumps straight into playing songs I've previously liked.

There's also a URL case for when someone references a link from your app. Like: "Play that playlist Glow sent me." Check out the documentation for the full set of AudioSearch criteria.

Sometimes people aren't asking Siri to take action, they just want to find something. When I ask: "Show me running playlists in CosmoTunes." Siri can display a list of entity search results. That's a nice default. But I spent a lot of time crafting the app's own search experience, and I'd love to show these results there.

To do this, I'll adopt the system .searchInApp schema.

The .system search schema introduced in iOS 17 is now named .system.searchInApp. It is part of the System App Schema domain, and it lets people search in your app with Siri, no matter which other domains you adopt, and even if you don't index your entities.

Siri calls this intent with the same string it searched for, and the intent's perform method finds and shows those results in the app.

Spotlight and structured search let Siri reason about your content. That's great if people ask Siri to play content in the app by name but just like in everyday conversations, people often refer to what they see.

That's why I want to allow people to interact with the audio content they're looking at on their screen.

Onscreen awareness is how your app connects what's visible on screen to structured information and actions the system understands.

Siri can then resolve references like: "Play the third one" or "that conversation" without the person naming them explicitly. When people start a Siri request, Siri has an understanding of text on screen, but it's limited to exactly what's in the pixels. For example, Siri can't act on the tracks shown, and it may not be able to tell you about the artist because the artist isn't currently shown on screen.

Adopting onscreen awareness APIs provides Siri with additional context of what entities are on screen, and where they are on screen. This onscreen context means Siri can answer detailed questions about those entities, and take action on them.

When adopting onscreen awareness, the NSUserActivity and View Annotation APIs are where you should start.

With NSUserActivity, attach .userActivity to the view representing your primary onscreen content. Use the View Entity annotation when the entity is one item among many on screen.

Attach .appEntityIdentifier to each view that represents an entity. In CosmoTunes, AlbumView uses a View Entity annotation because both the album and the containing tracks are visible. NowPlayingView uses NSUserActivity because the screen is dedicated to the currently playing item.

NSUserActivity and View Entity annotations are enough when a screen has a handful of entities. But there are two more onscreen awareness APIs. The first is for lists and collections, where you display many entities at once.

Tracks in CosmoTunes are displayed in lists in a few views of the app.

Collection annotations help me avoid the overhead of attaching an annotation to every single row. Instead, the system fetches identifiers lazily, as it needs them.

Collection annotations also let Siri discover entities that have been selected and scrolled off screen. Per row annotations disappear as soon as the view leaves the view hierarchy.

In SwiftUI, use the .appEntityIdentifier (forSelectionType:) modifier on a List, returning the EntityIdentifier for each item's selection ID.

The second API is the custom canvas view annotation. I built this custom canvas view that looks like a piano roll. It illustrates the notes in the current track and brings the unique retro look CosmoTunes is known for.

I want people to be able to act on the associated song using Siri whenever this canvas is visible.

To help the system understand this non-standard subview, I used the custom canvas view annotation. If you're using SwiftUI, check out how I adopted this in the PianoRollView in the CosmoTunes sample code.

UIKit and AppKit also support all of the onscreen awareness APIs.

Check out the documentation for: AppEntityAnnotatable, UICollectionViewAppIntentsDataSource, and appEntityUIElementProvider. And to learn more about how these entity annotations help power contextual menu items in UIKit apps, check out, "Modernize your UIKit app." After adopting onscreen awareness, some of the app's views show many entities at once.

Siri needs to quickly understand if the on-screen entities relate to a request. For example, someone asks Siri to play the third one. If Siri can't understand my on-screen entities quickly enough, it may ask to clarify or play something else entirely.

People can abandon the request when that happens. The entity display representations you customized earlier can help.

In CosmoTunes, I enabled display representation querying on the playlist entity query by implementing the displayRepresentations method.

Now, when Siri is trying to understand the content on screen, it can query just the text representation of the entity and skip the overhead of fetching the full content from the database.

Onscreen awareness provides Siri with additional context when people are looking at your app. Beyond the UI, your app already integrates closely with other parts of the system. To give Siri even more context, you can connect entities to integrations that you already adopt, like user notifications.

With this added context, your app entities act as a universal language. They let Siri understand not just what's on screen, but how other system integrations and time-sensitive events relate to your content.

I'll add entities to three integrations the apps already use: UserNotifications, NowPlaying, and AlarmKit.

When I'm done, I'll be able to say: "Play the live version" letting me easily switch to a different version of the currently playing song.

When a UnicornChat notification is announced on AirPods I can say: "Reply, 'ok, I'll swing by the unicorn supply store and pick those up.'" And Snooze it to snooze an alarm from CosmoTunes.

All three use the same pattern, and we call these entity annotations.

Annotating notifications with entities gives Siri concrete entity context when announcing a notification on AirPods. When listening to the announced notification, the person might want to act on the entity behind it, like replying to a message or checking off a reminder.

To give the additional context to Siri about what entity is associated with the UnicornChat notification, I'll update the posting flow. After importing AppIntents I assign the persistent message EntityIdentifier to the .appEntityIdentifiers property on UNMutableNotificationContent.

Note, that with the three entity annotation APIs I'm describing, you can't use TransientAppEntity. Transient entities are temporary model objects, so they don't have persistent identifiers. To add entity annotations to NowPlaying in CosmoTunes, I followed the same pattern. I'm already providing song attributes using MusicContent in the app's MediaSessionRepresentable conformance.

To enhance this state, I'll take the existing song, artist, and playlist entities, and add them to the appEntityIdentifiers property in order of most specific to least specific.

This enables contextual requests like: "Play the live version." With AlarmKit, I add a single EntityIdentifier to the appEntityIdentifier parameter on AlarmConfiguration when creating an alarm or a timer. With this, people can act on firing alarms and timers.

That's all it takes to connect your entities to notifications, Now Playing, and alarms.

We've covered several advanced ways to make your app work even better with Siri. As you think about next steps, a great place to start is by customizing your entity display representations. They are used to display your entities across the system.

From there, add your entities to the semantic index, and keep the index up to date, so Siri can always find your freshest content.

You might also consider making your entities accessible through Siri with an IntentValueQuery and in-app search. And annotating your views, activities, and your existing system integrations with entities can give Apple Intelligence even more context to work with.

When you're ready, look into donating UI interactions to help Apple Intelligence understand how people use your app, making for a more personalized experience.

To see any of these concepts applied, take a look at the sample projects.

For a hands-on look at adopting App Schemas, check out the "Code-Along: Make your app available to Siri." With the 27 releases, Apple Intelligence is transforming what Siri can do, and App Intents puts that transformative power directly in your hands. You now have everything you need to build delightful, elevated experiences that feel like a natural extension of the system. I can't wait to see what you create. Until next time.