# Evaluating multimodal prompts Measure how well an on-device model classifies food photos, using a multimodal prompt and a streamed image dataset. ## Overview A prompt is the input you give a model to guide its response. With the Foundation Models framework, a prompt can be multimodal: you add an image to a alongside your text, and the model reasons about the image and the text together. This sample code project is a macOS and iOS app named Grub Gander that: - Sends a food photo and a question to an on-device in one prompt. - Classifies the food as healthy, unhealthy, or not food at all. - Measures how well the model does in a separate evaluation. You run that evaluation as a test, using the Evaluations framework and a labeled image dataset. The scores cover overall accuracy, per-class precision and recall, and confidence calibration.  Image evaluation has a problem that text evaluation doesn’t: a decoded photo can use a large amount of memory. The built-in [`ArrayLoader`](/documentation/Evaluations/ArrayLoader) holds an entire dataset in memory at once, which is fine for text but doesn’t scale to thousands of images. To avoid that, this project streams its dataset from disk instead. ## Configure the sample code project The app and its evaluation use the on-device model designed for Apple Intelligence. Run the app on macOS 27 or later, or on a physical iOS 27 device. Use Xcode 27 or later. The evaluation dataset is large, so it isn’t included in the sample project. Prepare it before you run the tests: 1. Install the `uv` package manager if it isn’t already present. 2. From the `GrubGander/` directory, download the dataset into the test data folder: ```bash uv run Tools/prepare_dataset.py --out GrubGander/EvalData --per-class 100 ``` This download process writes an image-folder layout and an `eval.jsonl` manifest into `GrubGander/EvalData`. 3. Run the tests with Product > Test. The evaluation reads `EvalData` in place from the source tree, so there’s no resource-copy step to rebuild. ## Build a multimodal prompt The classifier returns a guided-generation result named `FoodVerdict`. The macro defines its shape, and its category is a `FoodCategory`. The project declares `reasoning` before `category` on purpose: the model fills the fields in order, so it commits to an explanation before it picks an answer: ```swift @Generable enum FoodCategory: String, Equatable, Sendable, Codable, CaseIterable { case healthy case unhealthy case notFood = "not_food" } @Generable struct FoodVerdict: Equatable, Sendable, Codable { @Guide(description: "A brief, specific explanation of why the image belongs to the chosen category.") let reasoning: String @Guide(description: "The single best category for the image") let category: FoodCategory @Guide(description: "How confident you are that the answer is correct.", .range(0...100)) let confidence: Int } ``` The `FoodClassifier.prompt(for:)` method builds a that pairs an image `Attachment` with the question text. Both the app and the evaluations call this method, so the prompt they send can’t drift apart: ```swift nonisolated static let question = "What category best describes this image?" /// Take an image and convert it to a prompt. nonisolated static func prompt(for image: CGImage) -> Prompt { Prompt { Attachment(image) question } } ``` Classifying an image-and-text prompt is no different from classifying a text prompt: the attached image is just another part of the prompt, and the instructions define the task. Two `classify(_:)` overloads run it: one takes a `CGImage` and builds the prompt; the other takes a prepared `Prompt`, opens a session, and returns the verdict: ```swift nonisolated static let instructions = """ You are a careful food-image classifier. Decide whether an image shows \ healthy food, unhealthy food, or something that isn't food at all. \ "Healthy" is whole or minimally processed food — fruits, vegetables, \ nuts, grains. "Unhealthy" is sugary, fried, or heavily processed food — \ desserts, candy, pastries. """ /// Classify an image on the on-device system model. nonisolated static func classify(_ image: CGImage) async throws -> FoodVerdict { try await classify(prompt(for: image)) } /// Classify a prepared prompt on the on-device system model. nonisolated static func classify(_ prompt: Prompt) async throws -> FoodVerdict { let session = LanguageModelSession(instructions: instructions) return try await session.respond(to: prompt, generating: FoodVerdict.self).content } ``` ## Stream the dataset one image at a time A [`Loader`](/documentation/Evaluations/Loader) supplies an evaluation’s samples. Instead of the built-in [`ArrayLoader`](/documentation/Evaluations/ArrayLoader), this project uses a `JSONLImageSampleLoader`. The loader: - Streams the dataset from disk. - Decodes each image only when the evaluation asks for it. - Keeps peak memory close to a single image. This custom loader conforms to [`Loader`](/documentation/Evaluations/Loader) and returns an `AsyncSequence` that produces samples lazily. Its `stream` reads a JSON Lines manifest one line at a time: ```swift struct JSONLImageSampleLoader: Loader { typealias Sample = ModelSample let directory: URL var maxPixelSize: Int = 1536 private struct Record: Decodable { let image: String let label: String } var stream: any AsyncSequence, any Error> { // Capture the loader's values so the producing task doesn't retain `self`. let manifestURL = directory.appending(path: "eval.jsonl") let imageRoot = directory let maxPixelSize = maxPixelSize return AsyncThrowingStream(ModelSample.self) { continuation in // A missing dataset yields an empty run rather than an error; callers // gate on `EvalDataset.isAvailable` and skip the eval before reaching here. guard FileManager.default.fileExists(atPath: manifestURL.path) else { continuation.finish() return } let task = Task { do { let handle = try FileHandle(forReadingFrom: manifestURL) defer { try? handle.close() } let decoder = JSONDecoder() // `.lines` streams the file lazily; it does not read it all up front. for try await line in handle.bytes.lines { try Task.checkCancellation() let trimmed = line.trimmingCharacters(in: .whitespaces) guard !trimmed.isEmpty else { continue } let record = try decoder.decode(Record.self, from: Data(trimmed.utf8)) guard let category = FoodCategory.fromDatasetLabel(record.label) else { continue // An unrecognized label, so skip rather than crash the run. } let imageURL = URL(fileURLWithPath: record.image, relativeTo: imageRoot) guard let image = Self.decodeImage(at: imageURL, maxPixelSize: maxPixelSize) else { continue // A missing or corrupt image, so skip it. } continuation.yield( ModelSample( prompt: FoodClassifier.prompt(for: image), expected: FoodVerdict(reasoning: "", category: category, confidence: 50), instructions: Instructions(FoodClassifier.instructions) ) ) } continuation.finish() } catch { continuation.finish(throwing: error) } } continuation.onTermination = { _ in task.cancel() } } } } ``` Two details keep peak memory to about one image: - `FileHandle.bytes.lines` reads the manifest lazily, so the whole file is never held in memory. - Each decoded `CGImage` lives only inside its sample’s [`prompt`](/documentation/Evaluations/ModelSample/prompt). Once the harness finishes that sample and moves to the next, nothing references the image, and the system frees it. Decoding uses ImageIO’s thumbnail path with a `maxPixelSize` cap, which downscales the image and applies its EXIF orientation without ever creating the full-resolution bitmap. The on-device model doesn’t need full-resolution photos. Each manifest line names an image and its *ground-truth* label, the answer that’s been verified as correct. The loader resolves the image paths against its `directory`: ``` {"image": "images/test/healthy/000123.jpg", "label": "healthy"} {"image": "images/test/not_food/004501.jpg", "label": "not_food"} ``` Only `category` has a ground-truth label; the expected `reasoning` is empty, and the expected `confidence` is a neutral 50. ## Evaluate the classifier The `FoodEvaluation` type conforms to [`Evaluation`](/documentation/Evaluations/Evaluation). Its `dataset` is a `JSONLImageSampleLoader` that points at the prepared `EvalData` folder, and its [`subject(from:)`](/documentation/Evaluations/Evaluation/subject(from:)) method routes each sample through the same `FoodClassifier.classify(_:)` call the app uses, so the evaluation measures exactly what ships: ```swift struct FoodEvaluation: Evaluation { let name = "Food Classification" var dataset: JSONLImageSampleLoader { JSONLImageSampleLoader(directory: EvalDataset.directory) } func subject(from sample: ModelSample) async throws -> ModelSubject { // Route through the exact call the app makes, so the eval measures what ships. let verdict = try await FoodClassifier.classify(sample.input.prompt) return ModelSubject(value: verdict) } } ``` The evaluation declares several [`Metric`](/documentation/Evaluations/Metric) values: one for overall accuracy; for each of the three ground-truth classes, a precision metric (the share of a class’s predictions that are correct) and a recall metric (the share of a class’s samples the model finds); and three more that track confidence overall, on correct predictions, and on wrong ones. Each [`Evaluator`](/documentation/Evaluations/Evaluator) scores one sample against one metric, calling [`ignore(rationale:)`](/documentation/Evaluations/Metric/ignore(rationale:)) to drop samples outside the class it measures: - A recall evaluator ignores every sample whose true label isn’t its class. - A precision evaluator ignores every sample the model didn’t predict as its class. The `recall(_:forClass:)` helper builds one such evaluator: ```swift private func recall(_ metric: Metric, forClass category: FoodCategory) -> Evaluator> { Evaluator { input, subject in guard input.expected?.category == category else { return metric.ignore() } return subject.value.category == category ? metric.passing() : metric.failing() } } ``` The confidence evaluators work differently from the precision and recall evaluators. They emit a continuous value with [`scoring(_:rationale:)`](/documentation/Evaluations/Metric/scoring(_:rationale:)) instead of a pass or fail, and they split confidence by whether the prediction was correct. Self-reported confidence is uncalibrated, so the signal worth watching is whether confidence drops on wrong answers. The [`aggregateMetrics(using:)`](/documentation/Evaluations/Evaluation/aggregateMetrics(using:)) method reduces the per-sample results into grouped summaries: the mean of the pass-or-fail accuracy metric gives the overall accuracy rate, and grouping keeps the per-class figures together. The test then prints the summary and computes a macro-averaged F1 across the three classes — a single score that combines each class’s precision and recall, and stays high only when both are high. Macro-averaging takes the plain mean of the three classes’ F1 scores, so each class counts equally regardless of how many samples it has. Finally, the test asserts conservative regression floors so a bad model fails: ```swift @Test( .enabled(if: SystemLanguageModel.default.availability == .available && EvalDataset.isAvailable), .evaluates(Self.evaluation) ) func runFoodClassification() async throws { let result = EvaluationContext.current.result print(result.groupedSummary) let macroF1 = Self.evaluation.macroF1(from: result) // Conservative regression floors so the eval can actually fail on a bad model. #expect(result.aggregateValue(.mean(of: Self.evaluation.correct)) >= 0.55) #expect(macroF1 >= 0.45) } ``` The `enabled(if:)` trait checks two things: the model’s availability and the dataset’s presence. The evaluation is skipped when either one is missing.  --- Copyright © 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)