Foundation Models: Is the .anyOf guide guaranteed to produce a valid string?

After experimenting with this code a bit I can see one potential issue: if you first initialize the SectionReader Tool with an empty list [] for sections, the .anyOf guide doesn't properly initialize and isn't enforced.

I can't tell from the above code when you are initializing SectionReaderTool, but here's an important rule:

When you initialize your LanguageModelSession with a tool, e.g. let session = LanguageModelSession(tools: [SectionReaderTool(.....)]), the property parameters will be computed at that time and the schema it produces will be added to the LanguageModelSession's instructions. Once the tool definition is in the instructions, it's never updated.

What that means, is if you end up changing your SectionReaderTool or sections at any time after that, the model will never see your updates. So in theory, if you sometimes were to have an empty list [] initially for the tool, the anyOf would never work, even if you later added sections to the list.

If that's not your issue, let me know and we can keep debugging :)

Thanks, @Apple Designer. I greatly appreciate any help you can give me.

I think this is not the issue because I initialize the Tool at the same time I initialize the session, and the article/sections are not supposed to change for the lifetime of the session. So, I do this:

let tools = [SectionReader(article: article, sections: articleSections)]
let session = LanguageModelSession(tools: tools, instructions: prompt)

So, I believe it should be fine for the parameters property to be computed at initialization time (the sections are known at initialization and do not change).

Got it, thanks! That narrows down the issue. Next, could you possibly share an error case example of an incorrect section the model passed in, as well as what the correct section options were at that time.

Any example data you have of this error will help us dig deeper to diagnose any potential bugs in the framework itself that could be causing the enum enforcement to fail. In the meantime, we'll start investigating, but data will definitely help. Thanks!

I'm dealing with encyclopedic articles, and quite often the model will request the "Introduction" section, even though no such string appears in the sections array. Sometimes it also hallucinates sections that it thinks ought to exist based on the prompt.

While I was debugging this, I constructed a simpler toy example. It might be easier if we talk about that instead. Here's a playground I made to demonstrate the issue:

import Playgrounds
import FoundationModels

#Playground {
    
    struct CityInfo: Tool {
        let validCities: [String]
        
        let name: String = "getCityInfo"
        let description: String = "Get information about a city."
        var parameters: GenerationSchema {
            GenerationSchema(
                type: GeneratedContent.self,
                properties: [
                    GenerationSchema.Property(
                        name: "city",
                        description: "The city to get information about.",
                        type: String.self,
                        guides: [.anyOf(validCities)]
                    )
                ]
            )
        }
        
        func call(arguments: GeneratedContent) throws -> String {
            print(arguments.generatedContent)
            let cityName = try arguments.value(String.self, forProperty: "city")
            let cityInfo = getCityInfo(for: cityName)
            return cityInfo
        }
        
        func getCityInfo(for city: String) -> String {
            switch city {
            case "London":
                return "Some info about London..."
            case "New York":
                return "Some info about New York..."
            case "Paris":
                return "Some info about Paris..."
            default:
                return "Unrecognized city!"
            }
        }
    }

    let citiesDefinedAtRuntime = ["London", "New York", "Paris"]
    let tools = [CityInfo(validCities: citiesDefinedAtRuntime)]
    
    let instructions = """
    You are a travel guide. Your job is to pick a city for the user to travel to based on their requirements. Once you've picked a city you should provide some information to the user about the city and why it's a good choice. To help you, you can use the getCityInfo tool to get information about a city.
    """
    
    let session = LanguageModelSession(tools: tools, instructions: instructions)
    
    let response = try await session.respond(to: "I want to travel to a big city in China")
    
}

When I run this, it usually tries to request info about Beijing (the generated content is {"city":"Beijing"}). Or, if I change the prompt to "I want to travel to a big city in Japan", it will try to request info about Tokyo, etc. You might need to run it a few times to reproduce the issue.

My understanding is that this should not be physically possible with guided generation. So, I'm wondering if I've set up the GenerationSchema correctly?

Thanks! I'm able to replicate this bug even if I pass in a fixed statically defined GenerationSchema like this:

@Generable
struct Arguments {
    @Guide(description: "The city to get information about.", .anyOf(["London", "New York", "Paris"]))
    let city: String
}

func call(arguments: Arguments) throws -> String {
    print("Arguments are", arguments.generatedContent)
    let cityName = arguments.city
    let cityInfo = getCityInfo(for: cityName)
    return cityInfo
}

I can still get the model to produce "Beijing"

That makes me fairly confident it's a bug on our end, and nothing wrong with your schema. We'll get working tracking down the bug, thanks!

Thanks – I didn't even think to test that because I assumed it must be something to do with creating a dynamic schema at runtime.

Do you have any ideas about a workaround? Is there some other way of defining a schema at runtime with guided enum-like behavior?

Or do you have any ideas about a timeframe for the bugfix (I mean, is this an iOS 26.3 type thing, or an iOS 27 type thing)?

Sorry, I can't get you a timeframe yet since we're still tracking down the failure, and where exactly the failure is in our library, or operating system, or model stack determines our timeline.

The best work-around I can offer for now is just add a verification in your tool call itself. For example:

func call(arguments: GeneratedContent) throws -> String {
    print("Arguments are", arguments.generatedContent)
    let cityName = try arguments.value(String.self, forProperty: "city")
    let cityInfo = getCityInfo(for: cityName)
    return cityInfo
}

func getCityInfo(for city: String) -> String {
    switch city {
    case "London":
        return "Some info about London..."
    case "New York":
        return "Some info about New York..."
    case "Paris":
        return "Some info about Paris..."
    default:
        return "Not a valid city. City must be one of:\(validCities)"
    }
}

Of course, this is slightly different behavior than if the enum worked though --- instead of never calling the tool if you ask for a city in China, instead the model may call the tool with "Beijing" and the tool needs to give the model an error message "Not a valid city. The city must be one of these: ....."

Ultimately the model gets the same information, but it's just a tad more inefficient.

For the article example, try an error message that prompts the model to call the tool again with correct input, for example something like: "Section '(section)' does not exist in the article. You must try calling myToolName again for one of these valid existing sections in the article: (realSectionList)."