Posts under Machine Learning & AI topic

Post

Replies

Boosts

Views

Activity

Disambiguation when multiple entities match
When a spoken phrase could match several entities in our catalog — same region, similar names, or partial matches — who is responsible for disambiguation: Siri via App Schemas and entity resolution, or the app via EntityStringQuery returning multiple candidates? What’s the recommended UX pattern for ‘Did you mean A or B?’
5
0
178
Jun ’26
On Protocol Extensibility & Multi-Modal Data
The Foundation Models framework is adding built-in OCR and barcode reader tools this year . If we implement a custom backend using the Language Model Protocol, can we return complex multi-modal objects (like bounding boxes or segmentation masks) back to the agentic flow, or is the protocol currently limited to text-based responses? For the 'Phone a Friend' pattern, is there a standard way to pass 'privacy-preserving embeddings' instead of raw text when calling a third-party model to maintain a higher level of user data protection?
1
0
69
Jun ’26
On Agentic Testing & Accessibility
Since agents in Xcode 27 can now interact with the accessibility tree and screenshots, can we provide 'developer hints' in our code to help the agent distinguish between decorative UI and critical interactive elements during automated testing? Can the Evaluations framework be used to 'score' the efficiency of an agent’s navigation path through the app, helping us identify where our App Intents might be creating confusing or redundant loops for Apple Intelligence?
0
0
97
Jun ’26
In-app text input vs system speech paths
If users dictate into a standard TextField via the keyboard mic instead of a dedicated in-app record button, does that text still benefit from App Intents entity resolution and indexed entities — or is keyboard dictation a separate pipeline where we lose domain vocabulary unless the user invokes Siri directly?
0
1
54
Jun ’26
On Performance & Backgrounding
While we now know about the continued-processing.gpu entitlement for background tasks, is there a similar NPU-specific entitlement or priority flag to ensure that an on-device foundation model isn't preempted by system-level Apple Intelligence features while the app is in the background?
1
0
75
Jun ’26
On-device model capabilities, limits, and versioning
What is the context window of the on-device model (AFM 3 Core Advanced and the 3B Core), and how should developers handle prompts that exceed it — automatic truncation, error, or developer-managed chunking? For guided/structured generation into typed Swift values, what are the limits on schema complexity (nesting depth, enums, arrays, optionals), and what is the failure mode when the model cannot satisfy the schema? How deterministic and reliable is on-device tool calling under the Tool protocol — are there guarantees on argument validity, and a recommended pattern for validating/repairing tool arguments before execution? For the new image input: what are the constraints on resolution, image count per prompt, and formats, and does passing images change which device tiers or which model (on-device vs PCC) services the request? Since the on-device model ships and updates with the OS, how should developers detect the active model version at runtime and guard against behavioral drift between OS releases? Is there a pinning or capability-query API? What are the realistic latency and concurrency expectations on supported hardware, and is there a supported way to run multiple sessions or background inference without thermal/throttling penalties?
2
0
125
Jun ’26
Creating an in-universe AI computer in my app
Last year after Apple foundation models framework was introduced, I begin working on a separate test Playground project to see how to use the foundation model framework to create an AI computer in my app that only has knowledge of in universe content that comes from within my app. Now with the OS 27 updates released I’m going back to work on that. I believe I can use the on-device system foundation model framework comfortably because I don’t think there’s a lot of content in my app that the AI has to know about Do you have any advice for using instructions to tell the model to focus on only the knowledge boundaries from within my app universe or might there be new tools this year in using foundation models framework that might help me achieve the limited knowledge scope that I want the AI to recognize and respond to for my app users.
1
0
83
Jun ’26
Guidance Around PCC
If a developer is eligible for Private Cloud Compute and then crosses the threshold, what happens to PCC calls? Is there a paid program for PCC that you fall back on or does a developer need to already have built into their app another model ready in the wings to take over once that threshold is reached?
4
1
135
Jun ’26
Dynamic profile switching
When using Dynamic Profiles to switch between the on-device model and Private Cloud Compute mid-session, how is the context window reconciled — if I build up context on PCC (larger window) and then route a turn back to the on-device model, what happens to the entries that exceed the on-device window? — Divya Ravi, Senior iOS Engineer
1
0
122
Jun ’26
The standalone Siri app and cross-surface continuity
The new standalone Siri app keeps conversation history synced via iCloud across iPhone, iPad, and Mac. Can third-party content, results, or an app's agent surface appear inside the Siri app (e.g., as referenced sources or follow-up actions), and can the user deep-link from a Siri-app result back into the originating app with state intact? Is any conversation context from the Siri app exposed to a developer's intent when an action is invoked, so the app can act with the relevant context, and what are the privacy boundaries on that? When the same action is invoked from different surfaces (in-app, system Siri, the Siri app) and across synced devices, how should developers reason about execution location and idempotency to avoid duplicate side effects?
0
0
48
Jun ’26
Disambiguation when multiple entities match
When a spoken phrase could match several entities in our catalog — same region, similar names, or partial matches — who is responsible for disambiguation: Siri via App Schemas and entity resolution, or the app via EntityStringQuery returning multiple candidates? What’s the recommended UX pattern for ‘Did you mean A or B?’
Replies
5
Boosts
0
Views
178
Activity
Jun ’26
Siri without opening the app
Can App Intents perform authenticated backend calls (Bearer token in Keychain / App Group) and return structured results to Siri, or must execution always launch the host app first?
Replies
1
Boosts
0
Views
152
Activity
Jun ’26
On Protocol Extensibility & Multi-Modal Data
The Foundation Models framework is adding built-in OCR and barcode reader tools this year . If we implement a custom backend using the Language Model Protocol, can we return complex multi-modal objects (like bounding boxes or segmentation masks) back to the agentic flow, or is the protocol currently limited to text-based responses? For the 'Phone a Friend' pattern, is there a standard way to pass 'privacy-preserving embeddings' instead of raw text when calling a third-party model to maintain a higher level of user data protection?
Replies
1
Boosts
0
Views
69
Activity
Jun ’26
Privacy, personalization, and App Store expectations
We offer both cloud-based AI (subscription) and are exploring on-device Apple Intelligence features. What user profile data is appropriate to inject into on-device model sessions under Apple’s privacy guidelines, and how should apps disclose hybrid cloud + on-device AI in privacy nutrition labels and review?
Replies
1
Boosts
0
Views
109
Activity
Jun ’26
Summarization that must not hallucinate numbers
What’s Apple’s guidance for using on-device models to turn structured JSON (time series, metrics, units) into a one-line natural-language summary without inventing values?
Replies
1
Boosts
0
Views
77
Activity
Jun ’26
Using FoundationModels framework in Extensions
LLMs are renowned for using so much RAM. Does this mean we can't essentially use FoundationModels in extensions such as MessageFilterExtension? I assume the system kills the extension before we even get a response.
Replies
2
Boosts
0
Views
118
Activity
Jun ’26
On Agentic Testing & Accessibility
Since agents in Xcode 27 can now interact with the accessibility tree and screenshots, can we provide 'developer hints' in our code to help the agent distinguish between decorative UI and critical interactive elements during automated testing? Can the Evaluations framework be used to 'score' the efficiency of an agent’s navigation path through the app, helping us identify where our App Intents might be creating confusing or redundant loops for Apple Intelligence?
Replies
0
Boosts
0
Views
97
Activity
Jun ’26
React Native + native AI bridge
What’s the supported integration path for Foundation Models and Apple Intelligence from a React Native app — thin Swift native module, App Intents only, or are these features effectively Swift-first?
Replies
2
Boosts
0
Views
71
Activity
Jun ’26
In-app text input vs system speech paths
If users dictate into a standard TextField via the keyboard mic instead of a dedicated in-app record button, does that text still benefit from App Intents entity resolution and indexed entities — or is keyboard dictation a separate pipeline where we lose domain vocabulary unless the user invokes Siri directly?
Replies
0
Boosts
1
Views
54
Activity
Jun ’26
On Performance & Backgrounding
While we now know about the continued-processing.gpu entitlement for background tasks, is there a similar NPU-specific entitlement or priority flag to ensure that an on-device foundation model isn't preempted by system-level Apple Intelligence features while the app is in the background?
Replies
1
Boosts
0
Views
75
Activity
Jun ’26
Guardrails for Siri + App Intents"
With Siri AI and App Intents, we can surface content from apps. Can we add guardrails or configure what Siri can or cannot say when dictating content?
Replies
1
Boosts
1
Views
347
Activity
Jun ’26
On-device model capabilities, limits, and versioning
What is the context window of the on-device model (AFM 3 Core Advanced and the 3B Core), and how should developers handle prompts that exceed it — automatic truncation, error, or developer-managed chunking? For guided/structured generation into typed Swift values, what are the limits on schema complexity (nesting depth, enums, arrays, optionals), and what is the failure mode when the model cannot satisfy the schema? How deterministic and reliable is on-device tool calling under the Tool protocol — are there guarantees on argument validity, and a recommended pattern for validating/repairing tool arguments before execution? For the new image input: what are the constraints on resolution, image count per prompt, and formats, and does passing images change which device tiers or which model (on-device vs PCC) services the request? Since the on-device model ships and updates with the OS, how should developers detect the active model version at runtime and guard against behavioral drift between OS releases? Is there a pinning or capability-query API? What are the realistic latency and concurrency expectations on supported hardware, and is there a supported way to run multiple sessions or background inference without thermal/throttling penalties?
Replies
2
Boosts
0
Views
125
Activity
Jun ’26
Mixed languages and foreign proper nouns
If the user’s device language is French but they speak English, or they use one language for the sentence and another for proper nouns, how does Siri handle transcription and entity resolution? Do we need per-locale entity indexing, aliases, or can semantic indexing work across languages?
Replies
0
Boosts
0
Views
55
Activity
Jun ’26
Speech recognition with large, dynamic vocabularies
Our users speak proper nouns and domain terms (place names, product jargon) that change frequently. What’s the best practice for improving recognition accuracy: dynamic contextual strings, on-device custom language resources, periodic vocabulary sync, or something else in the current Speech APIs?
Replies
1
Boosts
0
Views
81
Activity
Jun ’26
Creating an in-universe AI computer in my app
Last year after Apple foundation models framework was introduced, I begin working on a separate test Playground project to see how to use the foundation model framework to create an AI computer in my app that only has knowledge of in universe content that comes from within my app. Now with the OS 27 updates released I’m going back to work on that. I believe I can use the on-device system foundation model framework comfortably because I don’t think there’s a lot of content in my app that the AI has to know about Do you have any advice for using instructions to tell the model to focus on only the knowledge boundaries from within my app universe or might there be new tools this year in using foundation models framework that might help me achieve the limited knowledge scope that I want the AI to recognize and respond to for my app users.
Replies
1
Boosts
0
Views
83
Activity
Jun ’26
RAG support
What kind out-of-box on-device RAG support exists in the foundation models framework? (vector DBs, embedding methods etc., agentic RAG hooks?)
Replies
0
Boosts
0
Views
69
Activity
Jun ’26
Guidance Around PCC
If a developer is eligible for Private Cloud Compute and then crosses the threshold, what happens to PCC calls? Is there a paid program for PCC that you fall back on or does a developer need to already have built into their app another model ready in the wings to take over once that threshold is reached?
Replies
4
Boosts
1
Views
135
Activity
Jun ’26
Dynamic profile switching
When using Dynamic Profiles to switch between the on-device model and Private Cloud Compute mid-session, how is the context window reconciled — if I build up context on PCC (larger window) and then route a turn back to the on-device model, what happens to the entries that exceed the on-device window? — Divya Ravi, Senior iOS Engineer
Replies
1
Boosts
0
Views
122
Activity
Jun ’26
Hobbyist Eligibility for App Store Small Business Program
As a hobbyist developer, I develop apps mostly for my own use, and want to try using private cloud compute. As per https://developer.apple.com/private-cloud-compute I have applied for the App Store Small Business Program. Am I eligible for this and for pcc access if I have no apps in the App Store? I do have one in Test Flight.
Replies
0
Boosts
1
Views
99
Activity
Jun ’26
The standalone Siri app and cross-surface continuity
The new standalone Siri app keeps conversation history synced via iCloud across iPhone, iPad, and Mac. Can third-party content, results, or an app's agent surface appear inside the Siri app (e.g., as referenced sources or follow-up actions), and can the user deep-link from a Siri-app result back into the originating app with state intact? Is any conversation context from the Siri app exposed to a developer's intent when an action is invoked, so the app can act with the relevant context, and what are the privacy boundaries on that? When the same action is invoked from different surfaces (in-app, system Siri, the Siri app) and across synced devices, how should developers reason about execution location and idempotency to avoid duplicate side effects?
Replies
0
Boosts
0
Views
48
Activity
Jun ’26