I'm trying to use Foundation Models to identify the things in an image. That part is easy and is working well. I'd like to also know where in the image the things are. This is where I'm hitting a wall. For example, if there's an image of a horse and a cow, I'd like to be told (even approximate) coordinates of where in the image the horse is and where the cow is. Bounding boxes are fine for my needs. (Any coordinate system will work because it's easy enough to convert from one to another) The LanguageModelSession consistently lists the items in the image and gives me bounding boxes for their location that are reasonable approximations of where the images are in relation to one another, but it will (usually, not always) completely fail at explaining where the objects are in relation to the image as a whole, which is what I need. What's more, the failures are not consistent. Sometimes it will tell me that all the images are in the top half of the image. Other times, it will blow up the location of one or more objects in the image to multiples of their actual size. I've tried asking the LanguageModelSession to output the locations in various coordinate systems: raw pixel numbers normalized position (0 ... 1) integer percent position (0% ... 100%) a few different attempts at "soft location" systems where I just ask the LLM to tell me if the objects are in the top left corner or in the center for instance Of these, the "soft location" gives more consistent answers, but nothing that is complete enough to be usable. Asking for raw pixels gives answers that are ALMOST usable, but the position rectangles it gives are often off by one or two times the width or height of the object or suffer from the issue of "bunching" all the rectangles into the top of the image. I believe that part of the problem I'm having is that FoundationModels must downsample the image before processing. It appears that it's downsampling to 896px for the longest dimension of the image. Even accounting for this, though, I get strange output. Yes, I have considered using VisionKit's GenerateObjectnessBasedSaliencyImageRequest. It works well for another part of my project, but it doesn't fit exactly the particular need that I have here. It gives me locations of objects but not what they are. FoundationModels gives me what objects are in the image but not their locations. It may be that FoundationModels just isn't going to give me an accurate enough location for the objects in the image. It's a LLM, not a ML model, after all. If that's the case, I'd appreciate if someone would verify that so I can stop barking up this tree. It just seems like it should be possible, and I keep getting results that are almost accurate enough to be useful to me. Any help at all would be appreciated. Below are the instructions and prompt I'm using. let session = LanguageModelSession( instructions: """ You describe images to help another AI model identify and label distinct objects. Identify the distinct foreground subjects — objects, animals, people, or things that stand out as individual items someone would point to and name. Be specific (e.g. "a black and white cow", "a red coffee mug", "a wooden chair"). For each subject, provide a tight bounding box as pixel coordinates: - topLeft: upper-left corner of the box (x from left edge, y from top edge) - bottomRight: lower-right corner (x and y must be larger than topLeft's) - (0, 0) is the top-left pixel; x increases rightward, y increases downward - the exact pixel dimensions of each image are stated in the prompt you receive Also note background objects — items visible in the scene but not the main focus. Describe the setting — the background environment (surface, room, landscape, or space). Do not merge subjects and setting. A cow standing in a field has the cow as a subject and the field as the setting — not both as subjects. """ ) let prompt = Prompt { "Describe this \(imageWidth)×\(imageHeight) image. Bounding box coordinates are in pixels: (0,0) is top-left, (\(imageWidth),\(imageHeight)) is bottom-right." Attachment(modelImage.cgImage, orientation: modelImage.orientation) }

In the Foundation Models framework utilities package, the private method buildURLRequest in ChatCompletionsLanguageModel handles the construction of OpenAI-compatible API URLs: private func buildURLRequest(for request: ChatCompletionRequest) throws -> URLRequest { let isVersioned = baseURL.pathComponents.contains("v1") let endpoint = isVersioned ? "/chat/completions" : "/v1/chat/completions" let url = baseURL.appendingPathComponent(endpoint) ... } Problem The current implementation hardcodes "v1" to determine if the baseURL already includes a version. This limits compatibility with API providers using alternative versioning schemes. For instance, Volcengine Ark uses "v3" in its Base URL, making it difficult to seamlessly integrate their services. #Playground { let baseURL = URL(string: "https://ark.cn-beijing.volces.com/api/v3")! let modelName = "doubao-seed-2-0-mini-260428" let headers: [String : String] = [ "Authorization" : "Bearer \(apiKey)" ] let model = ChatCompletionsLanguageModel(name: modelName, url: baseURL, additionalHeaders: headers) let session = LanguageModelSession(model: model) do { let result = try await session.respond(to: "Hello").content } catch { print(error.localizedDescription) // HTTP error with status code 404: } } #Playground { let baseURL = URL(string: "https://ark.cn-beijing.volces.com/api/v3/responses")! let modelName = "doubao-seed-2-0-mini-260428" let headers: [String : String] = [ "Authorization" : "Bearer \(apiKey)" ] let model = ChatCompletionsLanguageModel(name: modelName, url: baseURL, additionalHeaders: headers) let session = LanguageModelSession(model: model) do { let result = try await session.respond(to: "Hello").content } catch { print(error.localizedDescription) /* HTTP error with status code 404: {"error":{"code":"InvalidAction","message":"The specified action is invalid: /api/v3/responses/v1/chat/completions Request id: 021784381168842fdfd2e3c33d5b6eddad55ac385080e727cab08","param":"","type":"NotFound"}} */ } } Suggested Solution To better accommodate different versioning conventions (e.g., v2, v3), we can leverage Swift's modern Regex (#/v\d+/#) to dynamically detect the version pattern in the path components. Here is a recommended update for the isVersioned check: let isVersioned = baseURL.pathComponents.contains { component in component.wholeMatch(of: #/v\d+/#) != nil }

I'm trying to make a third-party app's on-screen image available to Siri / Apple Intelligence so the user can say something like "send this to " and have the image handed off. And I'd like to confirm whether I'm using the intended mechanism or whether this particular case just isn't supported yet. What I'm trying to do My app shows a single image (it owns no photo library, the image is an in-app render). I want the user to be able to reference it as "this" and have Siri move it to another app. What I implemented (following Making onscreen content available to Siri and Apple Intelligence and WWDC26 240/343) A plain AppEntity with a stable id, conforming to Transferable, annotated on the image view with the iOS 27 .appEntityIdentifier(_:) modifier: struct OnScreenImageEntity: AppEntity, Transferable { static let typeDisplayRepresentation = TypeDisplayRepresentation(name: "On-Screen Image") static let defaultQuery = Query() let id: String static var transferRepresentation: some TransferRepresentation { DataRepresentation(exportedContentType: .plainText) { … } // distinctive text DataRepresentation(exportedContentType: .png) { … } // the image } struct Query: EntityQuery { func entities(for ids: [String]) async throws -> [OnScreenImageEntity] { logger.notice("entities(for:) CALLED \(ids)") // <- never fires return ids.filter { store.isCurrent($0) }.map(OnScreenImageEntity.init) } } } // on the view: Image(uiImage: image) .appEntityIdentifier(EntityIdentifier(for: OnScreenImageEntity.self, identifier: id)) What I observe (iOS 27 Beta 3, device) "Describe this image" / "Create a note for this" appear to use an automatic screenshot — the responses reference my app's UI chrome, and EntityQuery.entities(for:) is never called, so my entity's Transferable is not involved at all. "Send this to " doesn't attach the image; Siri says something like "I can't attach the image directly from your screen," which again sounds like a screen grab. entities(for:) is not called. The only flow that resolves my entity is the ChatGPT hand-off: asking Siri about the on-screen content calls entities(for:) (several times). But even there it stalls — Siri responds "could not clarify what you mean by 'about this'… document, image, or topic?", and no Transferable representation is ever read (my export closures never run). So the entity resolves, yet the content is never actually transferred. The older NSUserActivity.appEntityIdentifier route was never consumed at all; only the iOS 27 .appEntityIdentifier(_:) view modifier produced any resolution, and only in the ChatGPT flow above. I also understand from WWDC26 240/343 that the cross-app content move is built on IntentValueRepresentation (entity ↔ a system value type like IntentPerson), which the recipient imports via IntentValueQuery / IntentValueRepresentation(importing:). Since there doesn't seem to be a system value type for an arbitrary image (and IntentFile isn't one), I'm not sure an image entity can participate in that transfer at all. My questions Is .appEntityIdentifier(_:) the intended way to expose a single on-screen item in iOS 27? It only resolves my entity in the ChatGPT hand-off, and even then the Transferable is never read — under what conditions is the resolved entity's Transferable actually consumed? Are on-screen content requests ("describe this", "create a note for this") ever meant to use an app-provided AppEntity + Transferable, or are they always served by screen capture? (For me they never call entities(for:).) Is there a supported path for Siri to send an image from a third-party app's on-screen content to another app? If the transfer requires IntentValueRepresentation and there's no image system value, is a Transferable image representation ever consumed for this — and if so, how is it triggered? If this is simply not supported for image content yet, that's useful to know too, I just want to make sure I'm not missing a required piece (a schema conformance, an eligibility flag, a different annotation API, etc.). I have a minimal sample project that reproduces this (single annotated image, no photo library, logs to a subsystem so you can see entities(for:) never being called) and filed it as Feedback FB23813341 — happy to share details. Thanks!

Hi, so I just installed the 27 iOS beta and I use an iPad 4th generation or 10th and it was like a long time to start the update, but I’m OK with it. I just wanted to the Apple know if y’all could please give me the new Siri because last time when Apple Intelligence came I tried to install that update and it didn’t let me so I’m not forcing you that you’ll need to give it to me right now, but I was just seeing if y’all could please give it to me because I really need it. I don’t really need it, but I just want to test it out and for smarter use thank you Apple. If you have any questions, just text me on iMessage or reply to me by mail. Anything that you can do possibly but please approve me thank you.

I’m currently using ShowInAppSearchResultsIntent to open the app in a selectable view with the search results. The user can choose which view to pick via a @Parameter. Though with iOS 27 I can’t compile this intent anymore, because of this error: 'ShowInAppSearchResultsIntent' must only have a 'criteria' parameter What is the best practice to offer the same selection of the search view with iOS 27 and newer then? My Code import AppIntents @AppIntent(schema: .system.search) struct SearchAppIntent: ShowInAppSearchResultsIntent { static let searchScopes: [StringSearchScope] = [.general] var criteria: StringSearchCriteria // MARK: Parameters @Parameter(default: .loans) var target: SearchView? // MARK: Action Text static var parameterSummary: some ParameterSummary { Summary("Search \(\.$criteria) in \(\.$target)", table: "Shortcuts") } // MARK: Action @MainActor func perform() async throws -> some IntentResult { switch target { case .general, nil: // Open app search tab NavigationManager.shared.openAppSearch(with: criteria.term) case .contacts: // Open contacts tab NavigationManager.shared.openContactsSearch(with: criteria.term) } return .result() } }

I can't figure how to install it Here's google post: You can use the following free options directly inside Xcode 27:1. Built-in On-Device Predictive Code CompletionApple provides a free on-device, on-chip model that runs entirely locally on your Mac.Cost: 100% Free (no internet connection or subscription required).How it works: It uses Apple Silicon to predict and autocomplete your Swift code instantly as you type.Setup: Go to Xcode > Settings > Intelligence and ensure local code completion is toggled on.2. Free-Tier Cloud Models (ChatGPT, Claude, & Gemini)Xcode 27 explicitly features a native two-tier intelligence system. For simple completions, it uses your local chip. For complex planning, multi-turn conversations, and writing autonomous unit tests, it integrates directly with cloud providers. You can utilize the free tiers of these services:Anthropic Claude: You can generate a free API key from the Anthropic Developer Console to power Xcode 27’s coding agents.OpenAI ChatGPT: You can hook Xcode directly into OpenAI's free-tier API allowance.Google Gemini: Xcode 27 natively supports Google's ecosystem, allowing you to use a free Gemini API key.Setup: Navigate to Xcode > Settings > Intelligence, select your cloud provider, and paste your free API key.3. Fully Local Open-Weight Models via OllamaIf you want to handle complex agent tasks without data leaving your Mac, you can connect Xcode 27 to local open-source models. This requires an Apple Silicon Mac.Recommended Models: qwen2.5-coder (highly recommended for Swift and SwiftUI) or llama3-coder.Setup:Download and run Ollama.Pull the model via your Mac terminal (ollama run qwen2.5-coder).Use an Xcode 27 compatible local-host bridge tool or local API endpoint under the "Custom Provider" option in Xcode's Intelligence settings to link Ollama's local port (localhost:11434) straight into your workspace.Xcode 27 Agent SkillsWhen using these models in Xcode 27, they will automatically ingest Apple's native Agent Skills (like the SwiftUI Specialist Skill). This means even a generic free model will receive Apple's optimized context rules to write better, modern Swift 6 code.Are you looking to use the model mostly for inline code autocompletion or for the new conversational agent features (like having the AI autonomously write tests and fix bugs in your workspace)? I can walk you through the exact setup steps for either.19 sitesXcode 27 Beta Release Notes | Apple Developer DocumentationOverview. Xcode 27 beta includes Swift 6.4 and SDKs for iOS 27, iPadOS 27, tvOS 27, macOS 27, and visionOS 27. Xcode 27 beta suppo...Apple DeveloperInside Apple Intelligence and Xcode: Special Presentation | WWDC26so today we're going to build something fun live on stage together but first can we'll give you a quick tour of Xcode. 27. all rig...49sYouTube·Apple DeveloperSwiftUI Best Practices, straight from Apple's Xcode 27 Agent SkillSwiftUI Best Practices, straight from Apple's Xcode 27 Agent Skill. Xcode 27 launched during WWDC 2026 and includes Apple's SwiftU...SwiftLeeShow all

I'm trying to get some clarity on how the new Siri deals with IndexedEntities and whether it's worth adopting, considering our app does not fit into any of the predefined domain schemas. In running some tests with the TravelTracking sample app, it seems the only way I can get Siri to show any of the referenced entities is by using the exact phrasing (or extremely close to it) in one of the donated shortcuts. If I ask Siri to "Find closest landmark in TravelTracking" produces a result from the App in the form of an app snippet. But, if I then ask it "Text the description to Jane", it seeds the text with something like, "Niagara Falls is located in North America", instead of what's in the description field of the entity. General questions about the indexed data fail to show any results at all in Siri. For example: "Show me some landmarks from TravelTracking" or "Find Mount Fuji in TravelTracking" produce no results, even though the landmarks are indexed. My original assumption was that indexing data from your app would make it available to Siri, but it only seems to show up in on-device search and not in conversation with Siri itself. So is it the case that such data is only available through a Siri conversation if either you can adopt a domain schema or create a shortcut and use very close to the exact phraseology? And in the case of the latter, you can't really act on the returned entities because basically all you get is what is shown in a snippet? Maybe the on-screen intelligence picks up something here (seems to), but nothing deeper, even if it is defined in the entity. I've put in a feedback request (FB23796681) for a general database domain with schema for common database operations. Perhaps something like this and way to describe record types to aid in understanding from the LLM would go a long way toward making Siri more flexible for agentic use? I can get Siri to do a lot of the things that were shown at WWDC, but that tends to make you think you can do similar things with other types of apps and when you can't because of the domain limitations, it's very frustrating and feels limiting. It seems the domain types fit the apps Apple ships with the OS (Mail, Photos, Notes, etc), but not other types of apps that don't fit that criteria. If I'm missing something here, any guidance would be appreciated.

Hello, I am trying to set up App Shortcuts with App Intents in my app, which has an accent (é) in the name. It seems that with an accented application name (e.g. "Démo"), shortcuts phrases are not recognized by Siri or with the "App Shortcuts Preview" tool in Xcode. public struct DemoAppShortcuts: AppShortcutsProvider { public static var appShortcuts: [AppShortcut] { AppShortcut( intent: OpenDemoIntent(), phrases: [ "Find the tests in \(.applicationName)", ], shortTitle: "Find tests", systemImageName: "location" ) } } With Siri, when saying the phrase "Find the tests in Démo", the shortcut is not launched I tried with the "App Shortcuts Preview" tool in Xcode, it does not match any Intent. (see screenshot) I set up App Name synonyms as a workaround but it seems to not always work. Has anyone encountered this problem ? Is there any other workaround ? Is this a bug with iOS 27 / Xcode 27 ? I filed a feedback FB23791964 with an Xcode Project

I don’t even see the waitlist of the Apple Intelligence program, i think it’s lacking instruction’s how to apply, when it comes out etc.

My device is Hong Kong version iPhone 16e with HK Apple ID. After updating to iOS 27 Beta 1, the "Apple Intelligence & Siri" menu is completely missing from Settings. Only the legacy "Siri" icon appears in that location. There is no option to join the "Try New Siri" waitlist, and none of the AI features are accessible. What I've tried (all failed): · Multiple reboots · Reset All Settings · Reset Location & Privacy · Toggling Siri language between English (US) and Traditional Chinese · Removing and re-adding keyboard languages Hardware eligibility: The device meets all requirements for Apple Intelligence (A17 Pro or later, 8GB RAM). The issue appears to be a system-level configuration loading bug that prevents the OS from recognizing the device as eligible, despite being a Hong Kong unit with a Hong Kong Apple ID. Feedback Assistant ID: FB23733608 Has anyone else with a non-US region device encountered this on iOS 27 Beta 1? Any temporary workaround would be greatly appreciated. Otherwise, I'll wait for Beta 2.

Unless I'm missing something, it seems like the quota usage information for the Private Cloud Compute model is too limited. You can tell if you've reached your quota or are below it. If you are below your quota, you can tell if you're approaching the limit, but what does this actually mean? Am I over 50%, 90%, 99%? It would be nice to have actual numbers in the quota. For example, I can see my token usage for a session. If an app could keep track of that versus the quota, you could come up with something way more useful for the user. Example: You have 100,000 tokens per month, this app has made 4 requests, that used a total of 5,000 tokens. If the user has used on 95,000 tokens of their quota so far, they know they can maybe make ~4 more requests from the app before the limit is reached, so they know to be careful with their usage. If they've only used 10,000 tokens of their quota so far, they know that have some breathing room and can use the feature more freely. The way the current system is designed, you have no idea at all. Adding real numbers (even percentages – if we can get usage percentages for the app as well), would really help in giving useful feedback to the user on their usage of PCC. Right now, everything is too vague.

I had a great run in the first decade of iOS development. Not so much since. I had 180k downloaded units in the last year - but I'm excluded from foundation models because I did well before 2015. That seems like an odd policy. Apart from anything else - it explicitly punishes long-term accounts... Lifetime downloads...

Watch and learn the road to our future of anyone’s growing business

We are evaluating various AI frameworks to use within our code, and are hoping to use some of the build-in frameworks in macOS including CoreML and Vision. However, we need to use these frameworks in a background process (system extension) that has no user session attached to it. (To be pedantic, we'll be using an XPC service that is spawned by the system extension, but neither would have an associated user session). Saying the daemon-safe frameworks list has not been updated in a while is an understatement, but it's all we have to go on. CoreGraphics isn't even listed--back then it part of ApplicationServices (I think?) and ApplicationServices is a no go. Vision does use CoreGraphics symbols and data types so I have doubts. We do have a POC that uses both frameworks and they seem to function fine but obviously having something official is better. Any Apple engineers that can comment on this?

Dear Apple Support, I would like to report a long-standing issue affecting Khmer text recognition in Live Text (Vision Framework/OCR). Based on my testing, this issue has persisted for more than two years, from iOS 17 through iOS 27 Beta 3, and is also reproducible on iPadOS and macOS. I would appreciate clarification on the following questions: Is Apple aware of an issue where Live Text (OCR/Text Recognition) incorrectly recognizes Khmer script as Thai script, causing copied text to become Thai characters instead of Khmer? Has this issue been officially logged as a bug within the Vision Framework or Live Text team? Since this behavior has remained reproducible from iOS 17 to iOS 27 Beta 3, why has it not yet been resolved? Is the problem caused by: automatic language detection, the OCR recognition model, the Vision Framework, or another component of Apple's AI pipeline? Does Apple currently have a dedicated OCR and language recognition model for the Khmer script, or is Khmer being inferred through another language model? Is there an estimated timeline for improving Khmer OCR and preventing Khmer text from being misidentified as Thai? Can Apple confirm whether this issue affects all products using Vision Framework, including: Live Text Photos Preview Screenshot OCR APIs provided to third-party developers? How can Apple work with the Khmer technology community to improve OCR accuracy and language support for Khmer? This issue is more than a simple OCR bug. When Khmer text is automatically converted into Thai characters, users lose access to the original text, developers receive incorrect OCR output, and it negatively impacts the digital representation of the Khmer language. For reference, I have documented the issue in detail here: https://app.notion.com/p/Inaccurate-OCR-Language-Inference-Khmer-Script-Misidentified-as-Thai-in-Vision-Framework-2d8a24f4ee6680fcbc49d989f8bb606f I hope Apple can investigate this issue and prioritize improving Khmer language support across Vision Framework and Live Text. Thank you.

During WWDC26 Keynote a second generation on-device model was announced with better speech generation capabilities. Is there a new API available for developers to generate speech?

Hello, I recently learned about Private Cloud Compute (PCC): https://developer.apple.com/private-cloud-compute/ I am currently using a standard Developer Program account, and it seems that I cannot apply for the program directly. Is there an alternative? Also, is there any additional fee for using this service? If I want to call PCC in the app, for example, using the following code: let session = LanguageModelSession( model: PrivateCloudComputeLanguageModel() ) Do I need to apply for a specific plan to ensure that my App is successfully published on the App Store and available to users? Thank you!

I am using the following code in my iOS application. #Playground { let session = LanguageModelSession() let response = try await session.respond(to: "List all states of USA.") print(response.content) } And I get the following error: The operation couldn’t be completed. (com.apple.SensitiveContentAnalysisML error 15.) I have turned off Apple Intelligence and turned on again. No use. I am using Xcode 27 beta 2. any ideas?

Hello. I have a problem with the FoundationModels adapter and the Apple-hosted managed asset pack via TestFlight. I have created an adapter that works fine locally by creating a model via (fileURL: URL) on a real device, but I cannot create a model using background assets by downloading the adapter via TestFlight. Every time I try to get an adapter, the creation of the adapter is interrupted by the compatibleAdapterNotFound error. The aar. archive i created using a special command - xcrun ba-package foundation-models package --adapter-path aurelius1.fmadapter --asset-pack-id fmadapter-aurelius1-9799725 --output-path ./aurelius1.aar --platforms iOS --on-demand\ after that, I replaced "OnDemand": null with "OnDemand": {} in the manifest so that the Transporter could send my archive to the App Store Connect. I followed all the recommendations in this topic - https://origin-devforums.apple.com/forums/thread/823148 ...but unfortunately unsuccessfully I would appreciate any help in solving this problem. here is the code that I use in my app -

Hello, I have seen a great improvement of siri. However, my job requires a lot of communication though different languages, as mostly of those languages is not my primary language, the apple intelligence writing tool was a revolutionary tool that I have used on my daily bases. The tool remains there but attached to Siri, which takes out the advantage to use a shortcut for proofread which automatically replaces the entire text. Right now I need to copy and paste on Siri and copy the answer and paste. In my point of view we could just put back the writing tool, please 😭 separately as the previous version was flawless. Thank you.