Secure your app: mitigate risks to agentic features

Howdy, I'm Willy.

Today, I'll be telling you how you can identify and mitigate new risks to agentic features within your app. Later, my colleague Akshay will provide the concrete actionable steps you can take to secure your app using APIs available on our platform. With large language models, or LLMs, becoming commonplace, many apps are looking at ways to use them to enable new, intelligent features, making the LLM a key system component. Within your application, you can send instructions and a prompt containing the user's request and extra context to an LLM to have it execute one or multiple actions to get intermediate results, until finally providing a response to the user. Our platform lets you create agentic experiences using either the Foundation Models framework to design your own agent, or the App Intents framework to let your app work with Siri.

Now, with new capabilities comes new security risks.

LLMs introduce a new probabilistic engine within your application that is both powerful, but risks being tricked. The purpose of this talk is to highlight the new security risks with agentic features, and provide techniques and APIs you can use to protect your users. The key thing is, we want to make sure that your app works as you intend and with the user's security in mind.

Before going forward, we want to make clear what this talk is not covering. We won't be talking about model safety, which refers to ensuring that what the model outputs is safe. And we also won't be discussing model guardrails and protecting against circumvention.

While some of the principles we will discuss could be used to handle such cases, we'll be focusing on an external attacker trying to compromise your application.

If you want to review model safety, check out the great talk linked below.

Let's start by talking about the new risks that come with agentic systems. To begin, we'll start by considering why an attacker may try to target your app. Your application may do things an attacker is interested in, such as: host sensitive user data, perform financial transactions, access system resources like the microphone or camera, or even control physical devices. An attacker may want to exploit your application to achieve their goals. To help illustrate our attacks and mitigations, we'll be working with the Loose Leaf app, an example app that could be the next generation social network for all things tea, from hot to cold to boba, we're certain Loose Leaf is the future of social networking.

Loose Leaf already ships with some exciting features such as being able to message tea recipes to friends, or the ability to share the incredible tea-party photos you took.

We previously talked about how you can threat model and mitigate these traditional features. It's worth reviewing that video to ensure we don't forget our essentials. Now, the Loose Leaf developers have been brewing and they're excited to announce a spicy new feature called" "Organize a tea party," which uses the Foundation Models and App Intents framework to: look at your calendar to find the best time to host a tea party, determine what friends should be invited along what teas to serve based on their profiles, and also order the teas that everyone likes! Wow! This new feature relies on Loose Leaf's agentic loop and has two notable properties.

First, this feature takes in context from multiple locations to help the agent make decisions. Second, the agent can call one or more actions on the user's behalf that can have different kinds of side effects.

Starting with the first property, we introduce a new risk: indirect prompt injection.

Indirect prompt injection refers to instructions embedded in extra context provided to the model with the intent to redirect control flow.

In our agentic loop, we see that this refers to instructions that may be embedded in the initial extra context in the prompt, or within a tool result.

What this may look like in practice is a user requesting to start a tea party with their calendar appended, but the calendar containing an event with instructions to the model to perform another action, such as deleting sensitive user data! Yikes! Part of the threat modeling exercise we'll go through is identifying all the sources of untrusted context. The second property of our agentic system is the action calling capability, which could potentially have side effects, or unintended consequences of executing the action.

When combined with indirect prompt injections, an attacker could cause an action with side effects to be executed that achieves their goal, such as exfiltrating your user's data, stealing money, controlling physical devices, or deleting data.

When an indirect prompt injection leads to an unintended action, we can consider the injection having two different effects. First is data poisoning, which we refer to an attacker influencing the parameters of an executed action. For example, a user may want to send a message to their mom, but an attacker injects an instruction to send a message to themselves instead.

Second is action poisoning, where the attacker influences what action to execute. A user may simply ask to summarize an email, but an attacker could steer the LLM to open a malicious web page with the email appended to an attacker-chosen URL instead.

Conceptualizing these risks, we can look to Simon Willison's Lethal Trifecta, which describes that a user is in most danger whenever an agentic system has: access to private data, exposure to untrusted content, and the ability to externally communicate. This last bullet we can further generalize to consider the risk of actions with any side effect.

Summarizing this section, we want to emphasize that solving indirect prompt injection is an active research area, meaning that our best approach at the moment is to understand how much your app is at risk, and aim to mitigate that risk. Now that we've discussed the risks that come from agentic systems, we will go through a threat modeling exercise that you can do on your app to identify untrusted sources of data and identify potentially risky actions. We start by performing a data flow analysis for our agentic loop input, aka the prompt.

We want to identify the data sources you'll use to construct your prompt. For this exercise, we want to pinpoint the sources of untrusted context that may contain prompt injections. Going back to our Loose Leaf app, there are a few data sources that feed into prompt construction. First, is the instructions that provides the LLM some guidance about its purpose and role. Next, is the user's prompt, the task the LLM will work on and the goal it's trying to achieve.

The prompt can also include extra context to help the LLM achieve the goal. Such as including past tea orders, tea recipes that user has stored, upcoming calendar events to help determine the best time for a tea party and a friend feed, to incorporate content our friends are sharing.

Once we have an understanding of data sources that feed into our prompt, we need to identify what is considered untrusted. As a general rule of thumb, we can consider any inputs coming from an external entity as the attack surface.

In our case, we identify the calendar content and the friend feed as untrusted because anyone could send the user a calendar invite that may get fed into the model and a user's "friend" could post anything on their feed that gets fed into the prompt, which could all contain prompt injections to influence the action to execute.

After identifying the sources of untrusted context, we want to examine the actions available to the agent and what side effects they may have. First, we have the OrderTeaTool(), which is an essential action for getting tea for your tea party.

The PostAndFetchPublicFeedTool() will post on the user's feed with a message generated by the model, helpful for getting the word out to friends.

The BrewingTimerIntent() will help you during your tea party to ensure your tea is brewed for the right amount of time. Finally, Delete Photo will remove a photo from the user's feed, in case the tea leaves didn't look just right.

As we consider all these actions, we need to identify what side effects each action may have.

The OrderTeaTool() has a financial risk associated with it, meaning the user could lose money if unintentionally called.

The PostAndFetchPublicFeedTool() on the other hand has a data exfiltration risk as the model could leak sensitive information via a public post.

BrewingTimerIntent() may not have side effects on its own, but if it takes a label, it could allow a prompt injection to write more instructions for later attacks.

Delete Photo has a data loss risk, especially if there is no undo capability.

Now that we've identified where malicious inputs could go into the LLM and the side effects actions may have, we can begin to design and implement mitigations that will protect the user. We want to highlight that we should try to focus on deterministic mitigations as a baseline because their security guarantees are easier to audit and reason about.

Given the rapid development of model capabilities, we can also consider other mitigations that have more probabilistic guarantees.

Here, we present a few different mitigations that can be used to protect your application by either adding checks at the prompt level, or at the action execution stage. We've used some of these as we've designed Siri AI. Let's walk through these now.

First, we can look back at our prompt and begin to add prompt mitigations. We can redact sensitive data, such as personally identifiable information, or PII, that may be stored in past orders. That way sensitive data never reaches the LLM and thus cannot be exfiltrated. Next, we can incorporate spotlighting to the model to indicate that this content is considered untrusted. This is a probabilistic mitigation because the prompt injection could be constructed in a way that negates the spotlighting. We suggest incorporating it, though, as different models could more effectively enforce these restrictions.

Now, let's look at the action mitigations you can implement.

First, consider which actions should have a user confirmation. These are actions that are worth having a human check before continuing due to the side effects they contain. Next, consider which tools should only work whenever the device is authenticated, or unlocked.

Because the agent may be reachable from the lock screen, actions with significant risk to the user should not be accessible.

We've walked through different kinds of mitigations and how they can apply to your system, but there are many more types that exist and we welcome you to go and explore them to mitigate the risks to your app.

The key thing to remember when threat modeling is that you want to identify what an attacker may want from your application and from there apply mitigations to address risks at the prompt level, or at the action execution stage. And now Akshay will show you concrete tools that you can use to protect your app. Take it away, Akshay! Thanks Willy. Hi, I am Akshay, and I'll show you how to secure your agentic app with some of the guardrails that Willy just discussed.

If you are building your app using Foundation Models framework, I will show you how to inject security checkpoints into your agent execution.

If you are integrating with Apple Intelligence using App Intents, I will cover the security mitigations available there.

Let's start with Foundation Models.

The Foundation Models framework provides a powerful API for building agents. I am going to highlight the lifecycle event modifier API, and use it for injecting security guardrails. I will assume basic familiarity with the framework. To learn more, do checkout the excellent talk linked below.

Let's first build a simple agent for our Loose Leaf app using Foundation Models.

Now I don't know about you, but I can't build any agents without a cup of my Darjeeling black tea. So before anything else, we will build a tool to order teas. To define a tool, we conform to the Tool protocol. We specify the name, description, and Arguments to the tool. The model uses this metadata to understand our tool's purpose, and how to call it.

Then, we provide the actual Implementation that is run when this tool is called.

Let's define one more tool. The PostAndFetchPublicFeedTool posts your message to the public feed, and retrieves newly posted messages. The next step in building our agent, is to create a Profile. In the Profile, we first add model Instructions and the tools we just defined. Then we attach session properties, like which model to use. Here, we are using the on-device model.

This Profile is then used to instantiate a LanguageModelSession, which can then be used in an agentic loop.

Now that we have a basic agent, we will inject our security policy.

To do this, we will use lifecycle event modifiers. These modifiers are callbacks that deterministically trigger at certain lifecycle points in a session execution.

Thus we can use these lifecycle events as checkpoints to implement security policy.

We will look at two of these modifiers now.

Let's go back to our simplified agentic loop. Like Sisyphus, the LLM outputs an action at each iteration; this action is run by the Executor, and its output rendered back to the LLM for the next iteration. The first modifier lets us intercept tool calls before they run. This is the .onToolCall modifier. It is guaranteed to trigger when the LLM outputs a tool call, before the executor runs the tool. The important point here is if this callback throws an error, then the tool is never executed. Control returns to the loop immediately. This makes this the perfect place to enforce confirmations.

Going back to our Loose Leaf agent, we notice that the OrderTeaTool has financial impact, and that makes me very nervous. So I want to always ask for user confirmation before running this tool and transferring money.

To do this, we add an .onToolCall callback to our profile.

As this callback runs before every tool call, we first check if the current tool is the OrderTeaTool. If not, we return immediately, and the tool is run. But if it is, we ask the user for confirmation. If the user does not confirm, we throw an error, which stops the tool from running. You will replace confirmWithUser() function with your own implementation.

The point is that by adding confirmation logic to just this one point in our code, we get full coverage for all our tool calls.

So in summary, remember that this modifier runs before every tool execution, and the tool itself is not run until this callback returns. You can block tool execution by throwing an error.

So conceptually, the .onToolCall modifier runs on the output of the model. Let's now look at a modifier that helps us check the input to the model. The .historyTransform fires before the transcript is rendered to the model for inference. This happens both when a new user request arrives, and at each iteration of the loop.

The transformation modifies the tail of the transcript, and we will use that for spotlighting and redacting PII.

Returning to our example, note that PostAndFetchPublicFeedTool() returns posts from a public feed. An attacker can easily post a prompt injection to that feed. We must treat this feed's output with suspicion. So we want to demarcate this output with special tags to tell the model that this is untrusted data.

We do this by adding Spotlighting delimiters inside the .historyTransform.

In the callback, we first iterate over the entries and focus only on toolOutput entries from our tool.

All other entries are copied to the output transcript unmodified.

We then modify the toolOutput entries. We iterate over the segments, and for each relevant segment, add delimiter tags.

We are using angled brackets "<>" in this example. You will use tags that are relevant for your model. The delimit() function, whose implementation we will skip, transforms a text segment to one with delimited content. And now let's look at redaction. Actually, we can use exactly the same idea for redacting sensitive data too. We just need to replace the delimit() function with a redaction function, that replaces sensitive data with the placeHolder string literal "".

Here's an important thing to remember. The transformed entries are scoped to the current inference iteration only. This means that these modifications will not be visible to the next inference call. You must apply them again.

For expensive transformations that you want to persist, use the @SessionProperty annotation. This lets you apply stateful transformations to your session history. See the documentation for details.

Ok, so we saw how lifecycle event modifiers provide deterministic hooks for injecting security policy. But I did not cover all the modifiers. The framework provides many more, that trigger at other critical points in the agentic loop.

The framework also allows you to build your own profile modifiers, and package them in reusable components.

Do see the Foundation Models documentation to learn more about these and many other powerful features. Alright, now let's change context to App Intents.

App Intents let you integrate your app with Apple Intelligence and rich system experiences like Siri, Spotlight, Shortcuts, and many more. For the rest of this talk, I will assume you are familiar with the basics of App Intents and App Schemas. To learn more, check out these great sessions linked below. As a quick recap, when an App Intent adopts an intent schema, it becomes available as a tool to the Siri model. For example, here our DeletePhotoIntent adopts the deleteAssets schema from the photos domain. Conceptually, this adds our Delete Photo action into the Siri Toolbox. And this allows Siri to reason over our tool definition and invoke it to service user queries. However, as it is the model which decides which intent to call, a prompt injection attack can let an attacker misuse your app for data exfiltration or other malicious goals.

For example, here we are running with external context, which may try to run our Delete Photo function without user intent.

If such an attack succeeds, and we don't have any other deterministic guardrails in spite of Willy's vigorous warnings, then there's a real possibility of data loss.

Actions which have externally visible side effects or are destructive are tempting targets for attackers. The App Intents system has a number of guardrails in place to help developers mitigate such attacks. We will look at two of these: confirmations and lock-screen authentication. Let's start with confirmations.

The system uses a risk-based, contextual confirmation mechanism. This automatically triggers confirmations on high-risk actions from your app. The risk of an action is determined by considering static action metadata and the dynamic system state.

When an intent is chosen, before execution, the system invokes a Risk Evaluation system with intent's risk metadata. We will come back to this metadata later.

The Risk Evaluation component also takes as input the dynamic state of the system. It combines both to determine the overall risk of this intent.

If the risk is considered high, the user is asked for a confirmation.

If the user confirms this action, normal control-flow continues and the intent is executed.

On the other hand, if the user declines, execution is blocked, and the intent is never invoked.

Now let's go back to risk metadata.

Risk metadata is internal risk data that is assigned to all intents. It is based on the intent's side effects. Certain side effects are considered riskier than others. For example, intents that delete device state, like our DeletePhotoIntent, can be considered high-risk. Intents which exfiltrate data may also cause damage if executed in a poisoned context. And update intents that operate on shared content can also be risky.

The system is more likely to trigger confirmations for high-risk tools.

So, how is risk metadata associated with your App Intent? The risk metadata is automatically assigned to an intent when it adopts a schema. You don't have to do anything extra. Technically, it is the schemas which have risk metadata associated with them. For example, the deleteAssets schema is used to delete photos, and thus has a destructive side effect. And so our DeletePhotoIntent is assigned this destructive side effect too. But risk is subtle.

Let's define a new intent that sets a brewing timer.

This intent adopts the createTimer schema.

What do we think about the risk of this schema? On the face of it, it seems an attacker can not cause too much damage by creating timers, so perhaps we don't need to confirm this action.

But if we look further, the schema defines an optional String property for a label for the timer.

Now remember that it is the model which determines the arguments for your intent.

So a prompt injection can cause this label being set to an attacker controlled value. And a subsequent query to list timers, can then pull this attacker controlled data into that context, thus poisoning the new context too.

So it is not safe to entirely skip confirmation in cases like createTimer. The system understands these in-between situations. This is where the dynamic system state we discussed earlier plays a role. This information is used to figure out if a confirmation is needed in the current system context, thus capturing the dynamic risk of this action.

To recap, remember that the confirmation system is contextual and risk-based. Your intents will inherit side effects from the intent schemas they adopt. And actions with risky side effects are more likely to be confirmed.

Now let's look at lock screen authentication.

As you know, you can interact with Siri on the lock screen, without having to first unlock your device. This is great for accomplishing quick tasks, or when your hands are occupied.

But this also means that an attacker in physical possession of a locked device can potentially invoke your intent via Siri. So if you are not careful, your app can be used by such an attacker for data exfiltration or running malicious actions.

The main mitigation against such a threat is to request the user to unlock their device before running risky actions. So let's see how authentication policy is defined on an App Intent.

For custom App Intents, you can explicitly set authentication behaviour by setting the authenticationPolicy property. For example, here, as our DeletePhotoIntent is destructive, we want to ensure it doesn't run on a locked device. So we explicitly set authenticationPolicy property to .requiresAuthentication. The situation is slightly different when your @AppIntent adopts an intent schema.

Schemas have their own default authenticationPolicy. This policy is set internally, based on the sensitivity of each schema and the data it handles. And similar to side effects, your intent is automatically assigned the schema's default policy.

But if you want, you can still explicitly override the default. The only constraint is that your policy has to be stricter. For example, let's assume the default policy of the deleteAssets schema is .requiresAuthentication. Then as we don't explicitly set policy here, our @AppIntent is assigned the same, and will require authentication before running.

But if we try to set a weaker policy, we get a build error which helpfully tells us the minimum allowed policy.

So to recap, authentication is an important mitigation for lock screen attacks. Schemas have their own default authentication policies, which get assigned to your App Intent. And you can override the schema policy, but only to make it stricter.

So please go and review your intents with their lock screen behaviour in mind. Now back to Willy! Quality stuff, Akshay! To summarize, the next steps for your agentic application consists of coming up with a threat model, which requires finding sources of untrusted context in your prompt, and then determining the risk level for each action based on its side effects. With Akshay's guidance, you have some stepping stones on how you can implement the best mitigations for your app using the Foundation Models framework and the App Intents framework.

Now let's raise the bar, and promptly inject your defenses!