Hi,
I'm running SampleGenerator from the Evaluations framework with SystemLanguageModel.default in place of PrivateCloudComputeLanguageModel, otherwise as in the Book Tracker sample's BookSampleGenerator/main.swift. Same prompt, instructions, validator and 13 seed reviews.
Environment: Mac mini M5 Pro, macOS 27.2.1 (26B131), Xcode 27.2 (27B52), AFM 3 Core Advanced (8,192-token context).
Repro: equivalent to the Book Tracker sample's BookSampleGenerator/main.swift with PrivateCloudComputeLanguageModel() replaced by SystemLanguageModel.default and targetCount set to seeds + 20. I ran it from a test target rather than the sample's own target. Attached: the [SampleGenerator] log from one run.
let generator = SampleGenerator<ModelSample<BookTags>>(
Prompt("Generate diverse range of book reviews and corresponding tags. ..."),
samples: seeds, // the sample's 13 reviews
targetCount: seeds.count + 20,
sessionProvider: {
LanguageModelSession(model: SystemLanguageModel.default, instructions: generatorInstructions)
},
validator: { sample in /* the sample's four rules */ true }
)
for try await sample in generator.run() { /* ... */ }
1. The model receives the Prompt's debug description, not its text. The prompt segment in each session's transcript (and the [SampleGenerator] Prompt being sent to model: log line) begins:
Prompt(components: Swift.AsyncThrowingStream<FoundationModels.Prompt.Component, Swift.Error>(context: ...), synchronousComponents: Optional([FoundationModels.Prompt.Component.text(FoundationModels.Prompt.Text(value: "Generate diverse range of book reviews..."))]))
The examples that follow are also printed with String(describing:), including module and type names (MyModule.MySample(review: "...", tags: [...])).
2. Every request carries the full text of every sample so far. After the prompt comes "IMPORTANT: Generate N completely new examples with different starting prompts than any of these existing samples:", followed by every sample's full prompt text, seeds included, not just the selected examples. The request therefore grows with the dataset. On an 8K window the generator overflows, halves the batch, opens a new session, and finally logs Context window exhausted at batch size 1 twice in a row. Stopping. In every run that asked for 20 new samples (about ten runs), the dataset stopped at 15–22 samples in total, seeds included. run() then finishes normally, so the caller only finds out by counting.
3. With ModelSample<T>, the model is asked for every field, including instructions and TrajectoryExpectation. The on-device model fills them in. Every sample kept in two runs (9 of 9) had a writing brief in instructions ("Write a sweeping, reflective review…"), and 4 of 9 had invented tool-call expectations. In another run, all 6 samples had a brief where the review belongs ("Create 5 diverse book entries…"). The sample's validator passes these, since a brief is over 100 characters too. A custom @Generable type conforming to ModelSampleProtocol with only review and tags avoids this.
4. Deduplication is exact-match only. The model returns seed reviews with a sentence cut or added, and near-copies of its own samples with one word changed. Both pass. I check for shared 8-word runs in the validator instead.
5. Minor: samples includes the seeds, as documented, but the sample's main.swift prints allSamples.count as the number of new samples.
Questions:
- Is (1) intended, or should the prompt be passed through as text?
- Is
SampleGeneratormeant to support the on-device model? If so, could the existing-samples list be limited (to the selected examples, or by token budget)? - Could
run()report that it stopped short oftargetCount, for example by throwing or exposing a stop reason?