LLM search using Core Spotlight

Hi, I'm Jennifer, from the Spotlight engineering team. This year, we're taking search to a whole new level, with Foundation Models and Core Spotlight. You can build rich, conversational experiences in your app, simply by making your app content available to a large language model for reasoning and response generation.

Now I'm from California, and there are so many beautiful hikes in the area. I've been slowly making my way through some of the nicest trails, so I thought I'd build an app to help me along.

In my hiking trails app, I can already browse through state parks and trails. And once I've completed a trail, I like to write my own notes on what I enjoyed most about the hike. But it would be really great to be able to ask a language model all about the hikes I've already gone on, or even about new hikes I should try. Well, the Foundation Models framework makes it easy to get started. By introducing a language model session into the app, I can ask broad questions, and the model will answer just by drawing on its own knowledge of the world.

Now, I really only want answers about hikes that my app knows about.

And this is where Spotlight can help. The hiking trails app has indexed all these great hikes into a Core Spotlight search index.

So to help the model answer questions about those particular hikes, we can use the app's Core Spotlight search index, through tool-calling from the Foundation Models framework.

The Tool protocol from Foundation Models is a powerful concept that can be used to extend a model's capabilities, both by taking actions for a request, or by looking up context that a model needs to generate a response.

A tool works by declaring its arguments and output, along with some instructions on what the tool does.

And then, when the model decides it needs to use a tool, it will simply generate the arguments to call up that tool, and use that output for response generation.

If you haven't already, there are some great sessions, such as the "Deep dive into the Foundation Models framework", to learn more about how tool-calling works.

So what if we had a tool that lets a model generate a search on an app's Core Spotlight index? Well, today, we're introducing SpotlightSearchTool. It's a tool that adopts the tool protocol, to let a language model directly search your app's content in Core Spotlight for contextual response generation.

SpotlightSearchTool is available on iOS, iPadOS, macOS, and visionOS.

Before we get started, you'll want to make sure your app donates searchable content with Core Spotlight. Take a look at our past session on "Supporting semantic search with Core Spotlight", where we talk through how to donate searchable content to Spotlight, how to manage donations with a delegate and reindex extension, and how to perform structured search over item attributes, and search against the semantic index.

Once your app has donated searchable items to Core Spotlight, or indexed entities for Apple Intelligence, we're ready to begin.

We have a lot to cover in this video! We'll show you how to provide the new SpotlightSearchTool to your language model session.

Then we'll explore how to customize SpotlightSearchTool with guidance, knowledge providers, and specialized capabilities . And finally, we'll look at ways to evaluate model responses, with the evaluations framework. Alright let's get started.

First, let's see how SpotlightSearchTool can be used for contextual response generation.

In our hiking trails app, we've donated searchable items, to the Spotlight index that represent hiking trails. Each trail has metadata like the trail's name and location. And on some of the hikes, there's also some personal details such as the date when a hike was completed, and some notes I wrote about how those hikes went.

If I wanted to ask: What hikes have I gone on?, the model will need to search for items by attributes, like completion date and location, to be able to formulate its response. So let's build this functionality into our app. There's three things we'll look at when adopting SpotlightSearchTool. We'll need to configure the tool, for the kind of search we want the model to perform. Then we'll want to add additional context to the model, while the search is active, to get the best response. And finally, we'll explore different ways to display results in our app's user interface.

Configuring the tool is not too different from performing a Spotlight query directly. We'll start by importing both CoreSpotlight and FoundationModels. Then, in one line of code, the tool is ready to search your app's Core Spotlight index. You can also provide SpotlightSearchTool with a custom configuration. Here we're specifying a FileSource to perform a search against file paths in your app's sandbox.

Next you'll want to choose the right model for your app, whether it's the SystemLanguageModel or a model of your choosing, which you can do using the new Model Provider APIs.

Once you've chosen the model, add the new SpotlightSearchTool instance to your LanguageModelSession to start getting a response.

It feels like magic, but the response follows a path of tool calling and generation. For a question like: What hikes have I gone on?, the trajectory might start with the model deciding it needs to use SpotlightSearchTool the model will invoke the tool with a generated query Spotlight will execute that query and return a description of the result set back, and the model will reason over that output and generate its final response.

Now when I ask: What hikes have I gone on?, the model can generate an answer grounded in the app's content.

You might notice from some responses, that the model was not able to see all of the metadata, that was donated for the items.

That's because some metadata in the Spotlight index, like text content and HTML, is stored in a highly-compact representation that can be searched, but not recovered in a way that a language model can read it. For these cases, you'll want to consider providing additional metadata for an item, while SpotlightSearchTool is performing a search.

If your app donates searchable content to Core Spotlight, you'll already be familiar with the index delegate protocol.

Your app would set an index delegate on your CSSearchableIndex to handle reindex requests, such as when Spotlight needs to perform migration or recovery. For SpotlightSearchTool, we've added a method to the delegate to recover the full CSSearchableItem by its unique identifier. This allows the model to efficiently manage responses over potentially millions of results.

On your index delegate, simply adopt the new searchableItems (forIdentifiers:) to return the complete CSSearchableItem.

If your app has metadata that doesn't make sense to donate for search, but might be useful for the model to reason about, this is the right time to set any additional attributes on an item for the model to see.

Now that we've configured the tool to perform searches, we'll want to think about how to display results and responses in our user interface. The session response is a concise description over the result set. And in an assistant-style interface, this response is typically what an app would want to display.

But search results are also available directly on SpotlightSearchTool itself. For a list-style display, this is the best way to access searchable items, especially when the result set is large. Search replies pass back results in batches during the search, so query tokens can be used to manage the conversation stream, ensuring that user interface stays up-to-date with the model.

To access results from the SpotlightSearchTool, your app can wait for search replies and check for CSSearchableItem in the content of the reply. Search replies come as an async sequence of events, where each reply may include a batch of results, until the tool call completes.

Keep in mind that for any given response, the model may call SpotlightSearchTool more than once, before generating its final response. For that reason, use the queryToken on each reply, to determine when the user interface should refresh.

SpotlightSearchTool provides a host of search capabilities, from semantic search over text, to structured search over metadata, like dates, persons, locations and more. But depending on the language model you choose, you may want to customize SpotlightSearchTool both for the model, and your app content. There's a few ways to customize SpotlightSearchTool. Guidance profiles can be used to scope the tool's search capabilities. Providing the tool with world knowledge can help with reference resolution. And implementing custom pipeline stages, can improve model reasoning over your app's content.

SpotlightSearchTool provides its entire set of search capabilities to a model for guided generation. But guidance profiles can help scope that guidance to only what an app needs.

The hiking trails app doesn't donate person relationships, so guiding the model on how to search for authors and recipients, could be skipped for limited-context models. To selectively enable guidance on search capabilities like people and dates, use a GuidanceProfile.

You can even specify the exact list of metadata attributes, that the model should consider during a search.

Then set a dynamic guide level using the profile, when creating SpotlightSearchTool. On-device models have a more restricted model context size, so it's best to use focused guidance for simpler search capabilities.

Reference resolution is another way for your app to provide context that's not directly available in the search index. As an example, if the hiking trails app did donate person relationships, the person using the app might want to ask about other participants on the trail. In that case, the model needs to know who that person refers to in a prompt. If the app already knows who that person is, use a contact resolver to help the tool filter to the right set of results.

A contactResolver should return any contact information related to the user's identity, that can be matched against metadata in the search index.

And at last, your app can take advantage of custom pipeline stages, that take document reasoning even further. For really complex requests, the language model might forgo a simple search query, in favor of a pipeline search. A pipeline search brings together queries to the index, plus computation over a result set, for maximal efficiency. I could ask: how many trails have I hiked this year, and for each month, how many miles have I gone on average? Now, the model could perform a simple search and keep a tally in memory to answer the question. Or, if the result set is likely to be large, SpotlightSearchTool allows the model to request that Spotlight run a pipeline of search and computation stages.

With a pipeline search, the model can break down this complex query into a set of steps.

The model might generate a search for completed hikes, along with a counting stage that builds a table by month, then a stage that computes an average over all counts.

Pipeline stages allow the tool to perform efficient computation, or transformation, over a search result set on behalf of the model. And your app can participate by registering its own custom stages. Pipeline stages are Generable, so the model will generate a stage on-demand based on the user's prompt. And whenever a stage is generated, the model may choose to return data back to the app when it makes sense. The Foundation Models deep dive has a great segment on Guided Generation and Generable types that I highly recommend.

Let's take a look at the hiking trails app again. Some trails includes personal notes on how each hike went, so I might want to ask: I remember being really happy on some of my hikes. Which ones were they? On its own, the model could make its best guess at my happiness level, just by reading my notes.

Or, the app could register a custom stage, that computes a happiness score over each item, allowing the model to generate a response, solely on the computed top-scoring results.

To build a custom stage that computes a happiness score, we'll want to operate on CSSearchableItem as the input, and return a scored version as the output. The score could be computed by running a sentiment analysis model over the notes attribute on the item, or by some other custom logic, perhaps taking into account hikes rated with 5 stars. And since this is a Generable type, we can add properties with Guides to inform the model on which results to prefer. Then we simply register the stage by adding it to the tool's configuration. There's one more thing: remember how SpotlightSearchTool returns replies with search results for display? Well, the model may decide to send back a search reply with the output data of a pipeline stage, as another kind of partial result. From aggregate counts and tables, to free-form text or computed numeric values, your app can display some or all of these data types. And each reply comes with a handy LLM-generated label describing the content, giving your app the most flexibility for its user interface. With so many options for customization, from the model we choose and the searchable content our app donates, to guidance levels and custom reasoning, how can we verify, in a broad way, how well the model is responding in our app? Well, the Evaluations framework can help us in a few important ways. Not only can we quickly build evaluations to see how well the model is calling the tool, and how meaningful the response; we can also rapidly iterate on our app's searchable content paired with different guidance profiles on SpotlightSearchTool itself. The Evaluations framework has some great APIs for building an end-to-end evaluation suite, from large-scale dataset generation, to evaluation runs using custom metrics, and reporting. There are some great sessions that go in-depth on sample data generation APIs, and the video on creating robust evaluations for an agentic app is a great resource to get started on evaluating model responses with tool-calling.

For our purposes, we're going to focus on result coverage as a way to evaluate the hiking trails conversational experience. We want to know, given a dataset that's indexed in Core Spotlight, how well does the model generate responses based on the items we expect it to find. We'll start by defining a dataset that adopts the ModelSampleProtocol. Our TrailRequest already includes the natural language input that a person might ask about trails in our app, the output is a language model response and an expectation of the trajectory of the request. We'll also be adding a set of unique identifiers of searchable items that we expect the tool to return for that prompt.

If we have real data to test against, that's great; but if not, we can use Sample Generation APIs to generate data based on a prompt. Let's take a look at this in Xcode.

For our evaluations, we can define a set of hiking trails with the metadata that our app is expected to donate to Core Spotlight.

Then we'll build a set of seed samples to use in our evaluations. Samples can be serialized in any Codable format, and JSON works well for that purpose. Our samples include the query and the set of item identifiers we expect to be returned for the search. We can also provide a sample response that we can use later in a quality comparison with the model's actual response.

Using the Sample Generation APIs in a command line tool, I can expand this seed set to many more variations, to get broad coverage on how people might want to ask about trails.

The next step is to define our evaluation with metrics and trajectory. For our samples, we expect the trajectory of a response to include a call to SpotlightSearchTool to perform a query, so here's how we might define that expectation.

And here's an overview of an evaluation flow that takes into account how many expected items were included in the final response. In our test target, our evaluation will load the trail items and samples from our generated datasets. Then, we'll donate the trail items to Core Spotlight, and configure SpotlightSearchTool for this evaluation. Once the evaluation completes its run, we can set the expectation for any metric we've included, like result coverage.

This is just the start towards building comprehensive evaluations that will help you craft the best experience possible for your app. It's a big year for Foundation Models, and we hope you'll make the most of it. Download our sample code to see the hiking trails app in action. Try adding your own custom functionality to the app to really see what's possible. You might also want to add your own evaluation suite, with some inspiration from the evaluations agentic deep dive.

And remember, we're not writing search queries anymore. We're providing the content, and letting intelligence do the rest.