Hi everyone,
I've been chasing a stall in streamed guided generation that took me a while to pin down. I'm hoping someone here has seen it too, or can tell me whether it's expected.
My setup: macOS 27.2.1 (26B131), Xcode 27.2, SystemLanguageModel.default (AFM 3 Core Advanced, 8,192-token context), greedy sampling, with the schema included in the prompt.
What happens
streamResponse(to:generating:includeSchemaInPrompt:options:)delivers 16 snapshots in the first ~4 s, one per 10 output tokens.- Then nothing arrives for ~155 s: no snapshot and no error.
- At ~159 s one more snapshot arrives, and the stream immediately throws
LanguageModelError.contextSizeExceeded: "Content contains 8193 tokens, which exceeds the maximum allowed context size of 8192."
The prompt was 896 tokens, and the last snapshot reports 170 output tokens. That means about 7,100 tokens were generated without ever reaching a snapshot. The Foundation Models Instrument reports deltasCount: 730 for this response.
What the last snapshot shows
The field is a [String] of short bullets. In the final snapshot, the second bullet is a closed string: "The generalist used ". GeneratedContent(json:) accepts an escaped quote, single quotes, and a closed string followed by whitespace. It rejects a closed string followed by more text:
try GeneratedContent(json: #"{"justification": ["The generalist used "angry"#)
// Encountered content that cannot be completed into valid JSON
It looks as if the model wrote an unescaped " to start a quotation. That quote closed the string, and nothing after it could be parsed until the context window filled. Every freeze I could inspect stopped just before the model quoted someone ("The generalist's ", "The generalist stated "). None of the 199 finished responses I checked contained a double quote. On a run where the same input finished, the model wrote the word with single quotes ('angry').
The failure follows a single greedy path. In my app this input stalls on every cold request and finishes on every prewarmed one, while other inputs do the opposite. A standalone repro where this is the process's only request stalls with and without prewarm().
What I tried
- Bounding the array with
@Guide(..., .count(2...4))removed the stalls on the path my app uses: 0 of 34 panels and 0 of 27 end-to-end runs, down from 2 of 27. The same input still stalls on the other path. - The instruction "Paraphrase the panelists; don't quote them." removed the stalls, but it cost accuracy on another field.
- The instruction "If you quote a panelist, use single quotes." made things worse: 9 stalls in 68 requests instead of 1.
- My current workaround: if no snapshot arrives for 15 s, cancel the stream and show an error. After cancellation,
session.isRespondingwas false within a second.
Questions
- Is an unescaped
"inside a string field expected from guided generation? Or should constrained decoding prevent or escape it? - Once the output can no longer be parsed, could the stream fail right away instead of generating until the window is full?
- Is there a supported way to detect this on the client other than a timeout? And does canceling the consuming task reliably stop generation on the model side?
I've attached a self-contained repro (one Swift file, plus console output for the stalling run and the parser cases) to FB25591600. I'm happy to share more details or run anything that would help.
Thanks for taking a look!
Thanks for reporting the issue with your feedback report (FB25591600).
I'd say an unescaped " inside the string field (something like "The generalist used "angry" ...) is unexpected, because that is a syntax error for any JSON parser.
Ideally, the streaming should fail out once the output can no longer be parsed. Your observation that the generation process continues until exceeding the context window indicates that the underlying inference loop is completely decoupled from the snapshot parser, which I'd say is worth a feedback report, and you have already filed one.
Currently, the framework doesn't provide any way to detect the parser error: ResponseStream only yields when a new PartiallyGenerated snapshot is successfully decoded. The only way out I can see is the cancellation, as you have mentioned. Cancellation should reliably work, and your observation that session.isResponding turning false within a second confirms that.
Best,
——
Ziqiao Chen
Worldwide Developer Relations.