What signal should drive fallback for PrivateCloudComputeLanguageModel?

Question

Created 3w

Replies 4

Boosts 0

Views 1.2k

Participants 3

I'm building an app that uses PrivateCloudComputeLanguageModel as the primary inference tier with SystemLanguageModel as the fallback. The app is entitled (com.apple.developer.private-cloud-compute, granted and provisioned) and generations serve normally. My question is how a client should decide to fall back because in extended measurement, no public signal ever reflects the blocked state I actually hit.

What I measured (macOS 27.0 beta, 26A5416b / Xcode 27 beta 27A5237l, entitled signed bundle constructing PrivateCloudComputeLanguageModel directly):

Serving stopped mid-run with no leading signal: request N served normally (1.4 s), request N+1 threw LanguageModelError.rateLimited 494 ms later, at cumulative generation 786 for the day. 100% served → 100% refused between consecutive calls.
Every quota signal read healthy the entire time: before, during, and after the block. Across 1,517 readings in a single day: quotaUsage.status = belowLimit, isApproachingLimit = false, isLimitReached = false, resetDate = nil, availability = .available. A preflight on these APIs cannot see the condition.
The refusal is enforced locally after first contact: rejections return in ~230 ms vs ~0.9–1.4 s for served calls, so the client appears to cache the verdict rather than ask the server per-request.
The trigger is a cumulative ledger, not a request rate: 501 generations at 33/min in one 15-minute sitting was fine, and a later arm sustained 39.7/min; two bursts of 16 concurrent at 5.0 and 5.2 req/s served 32/32; the count that tripped survived a process restart and a 4.9-hour idle gap. But it's not a fixed daily number either. 501 fast was fine earlier the same day; the trip came 285 requests later. A rolling window on the order of hours-to-a-day is consistent with this, but nothing here measures its length.
Recovery: still blocked at +41 minutes (probes at +1/2/5/10/20/40 min all refused); fully recovered by +20 h with no intervention and no upgrade. Next day served normally from the first request.
quotaLimitReached never occurred: not once in ~800 generations plus the blocked period. The wall is typed as the transient error while carrying what the documentation describes as daily quota semantics ("a person either waits for their usage quota to refresh or they upgrade").
limitIncreaseSuggestion is presence-constant: nil at process start, non-nil on every reading after first PCC contact (identical while fully serving and while fully blocked) so its presence can't gate an upsell affordance.

The same signals-read-healthy-while-refusing divergence also reproduces against the developer-tool pool (fm serve), which I've reported separately (FB24273854 covers quota exhaustion surfacing there as a generic server_error/500 while /health reports the model available).

Questions:

Is attempt-and-classify the intended contract? Given that no preflight can observe the blocked state, should a client simply issue the request, treat the typed error as authoritative, and route to SystemLanguageModel? And is the ~230 ms local fail-fast on the blocked path contractual (cheap and safe to probe) or incidental? This is the one that decides how I ship; the rest are diagnostics behind it.
What does quotaUsage actually track, and at what granularity? I have driven the entitled app-tier path to a hard block and the developer-tool pool to exhaustion, and no field ever moved. Is there any consumption pattern that moves isApproachingLimit / isLimitReached / resetDate? If the intended answer is "only the per-person daily quota, which these volumes never approached," what is the wall I am hitting at ~786 cumulative, and why does it surface as rateLimited?
Should rateLimited and quotaLimitReached drive different client behavior — and which one is the daily allowance in practice? The documentation distinguishes rate limiting ("wait a period and retry") from daily exhaustion ("wait for refresh or upgrade"), but what I observe is the transient-typed error carrying the multi-hour ledger semantics. Concretely: what retry cadence is recommended after rateLimited (my measured recovery horizon was somewhere between 41 minutes and 20 hours. My current design stays on the on-device model and re-probes PCC at a low fixed interval rather than per-request)? And under what condition is resetDate ever populated, given it was nil even while blocked?
(Smaller, design guidance): my app can generate a few hundred requests as one feature batch (quiz generation over a user's imported document). Measured: 501 in a sitting was fine, cumulative 786 in a day was not. Since this allowance belongs to the person and is shared with every Apple Intelligence feature, is a several-hundred-request batch a reasonable use of it, or should features like this generate on demand? (I'm aware of the existing feature request for richer quota reporting (FB23378161); this is a narrower design question.)

I can attach the measurement driver and timestamped JSONL logs. The divergence is reproducible on a fresh day, though reaching the wall took ~800 cumulative generations.

Answer 1

Apple Designer OP

Apple

3w

Thanks @vocaro for reporting! I'm sending this to the engineers who worked on FB24273854, will either get back to you when I hear more, or they may respond here...

But sending all your measurements and logs for this new issue in a Feedback Report, along with the FB's auto-collected System Diagnostics definitely couldn't hurt.

We super appreciate all your FB reports! Those are absolute gold in the beta period ❤️

Answer 2

DTS Engineer OP

Apple

3w

I am not the mentioned engineer (who worked on FB24273854), but would like to be more clear on the question related to rateLimited.

@vocaro: Your observation seems to be that rateLimited, which is supposed to a transient error, carries “what the documentation describes as daily quota semantics”:

rateLimited was triggered by “a cumulative ledger”, and not how fast you made the requests.
Recovering from rateLimited may take up to 20 hours, or until next day.

If that is the case, the behavior is indeed worth a closer look. Would you mind to file a feedback report against that as well, with a sysdiagnose (and the details of your observation), and then share the report ID here? To capture a sysdiagnose, follow this instructions.

As my colleague mentioned, we really appreciate you spending time on filing the reports for us.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Answer 3

Apple Designer OP

Apple

2w

LanguageModelError.rateLimited(_:) docs say:

This failure can happen if you make too many requests in a short window. You can recover from this error by spacing your requests or reducing system load. The exact solution may be model dependent.

Since rate limiting is more about the current server load, it's up to each server model provider to decide when to rate limit. I wouldn't assume a fixed number, since it's not unusual for servers to take into account current capacity, per-request length, and other factors.

We'll investigate more on your report, however, this definitely feels like the culprit to me:

Measured: 501 in a sitting was fine, cumulative 786 in a day was not.

I would suggest trying to chunk your requests differently... instead of 500+ short requests, could you send <200 longer requests or pace them differently? My gut instinct is 500+ requests sounds like too many requests for a quiz-from-document task.

If you'd like to include a reproducible example of the kind of requests your sending in your new report, I can definitely take a look to give suggestions.

Answer 4

vocaro OP

2w

Thanks for the replies. I filed FB24515295 with the measurement driver, timestamped logs from both runs, and Feedback Assistant’s system diagnostics.

Ziqiao, you're right about the original rateLimited result, but to tighten up the recovery timing:

It followed cumulative usage, not request speed. The run sustained 39.7 generations/minute, and two bursts of 16 concurrent requests served 32/32 at 5.0 and 5.2 requests/second. The count also carried across a process restart and a 4.9-hour idle gap.
I didn’t actually observe it taking 20 hours to recover. Probes through +40 minutes all failed, and the next probe at +19.98 hours worked. So all I can really say is that recovery happened somewhere in the interval of (41 minutes, 19.98 hours].

After your reply, I deliberately ran the entitled app-tier driver to the wall again on the same macOS and Xcode builds. This time I got a pretty interesting contrast:

The 1,000-attempt run averaged 64.8 requests/minute.
The attempt right before the wall served in 840 ms.
The final attempt threw PrivateCloudComputeLanguageModel.Error.quotaLimitReached in 67 ms, not LanguageModelError.rateLimited.
Right before and after that error, quotaUsage still said belowLimit, not approaching, not reached, and resetDate = nil. Availability still said .available.

So this exact build can produce the expected quotaLimitReached error, but the earlier cumulative wall came through as rateLimited. The faster run also didn’t produce rateLimited, so request speed by itself doesn’t seem to explain the earlier result, though server conditions could’ve been different.

The sysdiagnose in FB24515295 was collected while this new quotaLimitReached block was active, so it captures the live quotaUsage and resetDate mismatch. I also attached the August 20 logs to the same Feedback report. Those cover the earlier rateLimited wall and recovery probes.

On the request shape... The hundreds of short, unique prompts are for diagnostics and development, not how I’m planning to ship quiz generation. I agree that the product should use a bounded, paced, on-demand, or on-device approach instead of firing off 500+ short PCC requests as one batch. I included the full driver so the framework team can see exactly what I sent.

If the team specifically needs a sysdiagnose from a wall that comes through as rateLimited, I can try again after the allowance refills. Since today’s same-build run returned quotaLimitReached instead, I can’t force which classification I’ll get. If there’s a logging profile or predicate I should enable first, let me know so the next run is as useful as possible.

Thanks!