Hi — I built SiriReady Bench, an external benchmark that measures whether an app's App Intents implementation is actually reachable the way people talk: static audit (App Schemas, IndexedEntity, View Annotations), a bilingual EN/JA utterance dataset (1,272 cases), and a real-run harness that drives built apps through AppIntentsTesting on an iOS 27.0 simulator (Xcode 27.1 / iOS 27.1 SDK).
Repo: https://github.com/NagaYu/siriready-bench
Dataset (dev split): https://huggingface.co/datasets/NagaYu/siriready-bench
Four reference apps (messages/calendar, correct and intentionally broken versions) were measured. Every number is labelled by source (real / mock / static). Utterance-to-intent arrival is a rule-based mock — AppIntentsTesting calls intents directly and never exercises Siri's language understanding, and I say so in the results. A few measured findings I'd like to sanity-check with the team:
-
Correct apps over-confirm. Even after a request names exactly one entity,
perform()still requests confirmation (56 cases in messages, 54 in calendar, in the reference implementations). Is "always confirm" the expected behaviour for these schemas, or is there a supported way for the system to skip confirmation when resolution is unambiguous? -
No tolerance for typos or sound-alike spellings, and index gaps.
spotlightQueryand entity string queries return nothing for misspelled or near-sound names. Possessive titles ("Jon's …"), conversation readings (Japanese yomi), and calendar aliases fall out of the index entirely — a Japanese calendar alias was not found and the event was created in the default calendar instead; designating a conversation by its reading returned 3 of 12 probes. Are readings/aliases intended to be indexed, and is fuzzy matching on the roadmap? -
The toolchain sets a floor. A messages app built without an entity string query does not compile: Xcode 27.1 raises a halting error requiring an
EntityStringQuery, so in real runs that "broken" app scored higher than a static reading suggests. Sharing this because it changes what "broken" can even mean for App Intents in iOS 27. -
Privacy routing is input-dependent. In the broken calendar app, an off-device call was observed in 24 of 24 location-bearing requests and in none of the other 84 (intercepted locally, never sent). A static privacy finding alone doesn't tell you when it fires.
Scores for the four apps (MOCK + REAL, real 54% · mock 25% · static 21%): messages_ok 95.0, messages_broken 68.8, calendar_ok 94.0, calendar_broken 83.4. Full tables, measurement conditions and the mock-vs-real divergence log are in BENCHMARKS.md in the repo. Physical-device Siri spot checks are protocol-only so far (not yet recorded) — also stated in the repo.
I'd genuinely like to know which of these behaviours are known/expected, and whether the team would read a Feedback report on the indexing gaps. The project is available for transfer or licensing; contact is via my portfolio (linked in the repo README).
— Yuta Nagao