FoundationModels guided generation: empty token masks and severe slowdowns on macOS 27 betas 5, 6 and 7

Thank you so much for the report with the sys-diagnose and the feedback update! Those are details essential for diagnosing this issue.

Apologies for the delay ---I've pinged the experts to check where we are with this one and will update here once I have an update.

Of course, your feedback case should update with new notes too soon, but I'll try to help push this one. :)

Yeah, I think it did dupe the post and that was on my end, sorry about that. Thank you for the support and please let me know if you need anything else : )

@boxbot 2 things:

1. Can you send us a Transcript (optionally also sample code) of any session that replicates the issue? The instructions are here: Inspecting session transcripts and reporting model feedback ... basically you export a session's transcript as a feedback attachment > grab the file > add it to your FB report.

2. Can you send the results of SystemLanguageModel.variant at runtime?

Kindly add these to FB24310823.

Why? We suspect this is model-specific behavior, but we need a session example with actual inputs and output to see exactly what you're seeing and replicate. The transcript will tell us exactly which model assets are at play, while the .variant will tell us if this is AFM 3 Core or AFM 3 Core Advanced causing the issue.

I have updated Feedback FB24310823 with the requested details and sample code as well.

Please let me know if you need anything more from my end.

Thanks Kevin Keheley

Thanks! I think we're getting somewhere, but just to make sure, 2 follow-up questions:

1. Could you send a specific example that you see running "slow" to the Feedback report?

2. Are you seeing this happen more often in requests that include tool-calling?

Alright, sorry for the delay response here. Some things going sideways at work. Anyway, updated the FB with the requested items. Please let me know if you need anything else...Thanks :)

Just wanted to follow up and make sure you had everything you needed from me at this. time?

thanks kevin

We've got everything to investigate. Thank you for your patience and all the follow-ups!

Yay, and now we have identified the problem and have a workaround for you until the fix lands:

1. You can safely ignore all those "Something is likely wrong with the tokenizer" logs... turns out nothing is wrong, the team says they're work on those logs to be more informative.

2. The issue is in guided generation. Until a fix lands, try adding (either to your prompt or instructions) "Respond in well formatted JSON."

"Respond in well formatted JSON." should boost the model just enough to overcome the current issue it's having.

But will it generate JSON? Nope. Your code will still generate your Generable types. This is just a model hack.

You guys rock. I will update my two apps and report back.

thank you very much :)

So tested this over the weekend, added the respond with well formatted JSON to the instructions (I have not added this to the prompt but can test that today if needed) and I am seeing the same results but might have found some addition details.

I have updated the feedback with a lot of details on the project we are building and what we found this weekend.

Thanks Kevin

Thanks for the updates! I've relayed those to our Generable experts who are working on the issue.

But another workaround worth a try:

Personally I've observed that when you encounter these issues on a long Generable task, meaning a "Big Generable" structure with a lot of properties, a lot of list types, or complex nested-Generable-types, the most reliable workaround I've tried is to break up the Generable.

For example, say you have a Generable to plan a game that should come up with the game title, description, characters, and locations.... the session stalls and fails, it's "too much" for the model with this current issue.

In this case, the solution is to break up task into an ordered sequence:

1. A Generable and session to create the game title and description.
2. Feed the results into session 2, which will create the locations, since those depend on the game's title and description.
3. Feed the cumulative results into session 3, which will create all the characters, since what characters make sense depends on the locations and game's descriptions.

That's an example where there's a clear dependency order, so you split 1 "Big Generable" into 3 Generable types (Game, Locations, Characters) across 3 sequential sessions.

Another pattern, (if it preserves the original meaning of your Big Generable) is to split, parallelize, and then combine at the end.

Based on the semantics of your Big Generable, essentially:

1. Sketch out a workflow (pen and paper or whatever's easiest) of sequential or parallelizable parts to split up the task across multiple sessions.
2. Based on your sketch, start with the 1st simplest split to implement in code.
3. Try it out --> now see after the 1st split, which parts are lagging or having bad results.
4. Continue to split based on the principles of least-effort and which sub-parts are performing worst.

Alright, I was thinking about this over the weekend for sure. Breaking it up to see if that helps at all. I also did not add the JSON to the prompt and was going to test that as well.

I will let you know once I update the FB with anything else I find. Mostly wanted to give you all some more deep context as to what I am doing here and how I am doing it to make sure you had the full picture!

I tested on beta 8 yesterday, updated some details in the FB on this issue.

Of note, we are breaking out the Generable into 5 (at least) currently. Think of it as each section of the App Review Guidelines. I have updated a breakdown in the FB on this and some more on the app and what it is doing under the hood.

Of note the on-device models are not doing a review so to speak but are being used as a judgement pass and also to generate things like developer discussion points.

Would the main suggestion be to break down the Generable even more to see if that could help level things out a bit?

Update on the macOS 27.0 release candidate, build 26A428, Xcode 27 RC.

We reran our fixed set of scans with the log stream capture. Nothing changed from beta 8. Every guided generation request still logs the tokenizer pairs (about 10 to 20 per second, same emitter and category), short requests still finish in 3 to 12 seconds, and long requests still crawl to about 160 seconds with the recognizer index above 50,000 before stopping, which we read as the context window filling. Three of five sections on our busiest project did that tonight, at 159.6, 160.2 and 157.8 seconds. The scan took 10 minutes 43 seconds; a healthy build does it in under a minute.

Two things worth adding to the record. First, on the same build the same Google Play instructions ran five sections in under ten seconds each on a project with little to say, so the prompt and the store are not the variable. Output length is. Second, the "Respond in well formatted JSON." line has been in our instructions since August 28 and made no measurable difference on the RC either.

I have updated the FB on this and added all the terminal outputs and details.

I also wanted to add that I was testing this on a MacBook Neo using the AFM 3 Core model. I am still seeing that token error in the terminal logs, but of note the MacBook Neo runs the app and more detailed scan faster, cleaner than a MacBook Pro using the AFM 3 Advanced model.

The Neo finishes before the Pro and does not add hallucinations as the MBP does.

Update, September 10, macOS 27.0 (26A428), two machines.

We ran the same Thorough scan of the same Android project with the same TestFlight build on a MacBook Pro (M4 Pro) and on a MacBook Neo (A18 Pro, 8 GB). Both on 26A428.

We read SystemLanguageModel.default.variant at runtime on each: the MacBook Pro reports AFM 3 Core Advanced with a context size of 8192, the MacBook Neo reports AFM 3 Core with a context size of 4096.

AFM 3 Core: five guided requests, 5 to 14 seconds each, whole scan 40 seconds, generation stops normally, zero reviewer considerations. The empty mask and allowedTokenIDs errors still scroll in the log, so the log signature is on both models.

AFM 3 Core Advanced: five of ten requests generated until the context window filled and threw the context size error at 159 to 207 seconds each. Whole scan about 15 minutes. The retried sections produced considerations with fabricated evidence.

Generation speed is about the same on both. The difference is that Core stops and Core Advanced does not. Every earlier capture on our Feedback was on the M4 Pro, so this is the first clean run we have seen on macOS 27 since beta 4, and it is the Core model.

Two questions for anyone who can answer.

Does this reproduce for others only on machines that get AFM 3 Core Advanced (M3 or later with 12 GB or more)?

And is there any supported way for an app to ask for AFM 3 Core on a machine that has Core Advanced, until the fix lands? The variant property is read only as far as we can see.

Details and both captures are on FB24310823.

Hi, we’re seeing a related @Generable issue: the output is schema-valid, but semantic associations can still be wrong — for example, a correct fact may be attached to the wrong source/entity.

In short: type-safe ≠ meaning-safe.

We now keep IDs, source refs, and provenance deterministic outside the model. Is this a known guided-generation issue?