Hello, I am developing an iOS app that uses machine learning models. To improve accuracy and user experience, I would like to download .mlmodel files (compiled and compressed as zip files) from our own server after the app is installed, and use them for inference within the app. No executable code, scripts, or dynamic libraries will be downloaded—only model data files are used. According to App Store Review Guideline 2.5.2, I understand that apps may not download or execute code which introduces or changes features or functionality. In this case, are compiled and zip-compressed .mlmodel files considered "data" rather than "code", and is it allowed to download and use them in the app? If there are any restrictions or best practices related to this, please let me know. Thank you.

Hello! I have a swift program that tracks the location of a ball (through the back camera). It seems to be working fine, but the only issue is the run time, particularly my concatenate, normalize, and argmax functions, which are meant to be a 1 to 1 copy of the PyTorch argmax function and the following python lines: imgs = np.concatenate((img, img_prev, img_preprev), axis=2) imgs = imgs.astype(np.float32)/255.0 imgs = np.rollaxis(imgs, 2, 0) inp = np.expand_dims(imgs, axis=0) # used to pass into model However, I need my program to run in real time and in an ideal world, I want it to run way under real time. Below is a run down of the run times that result from my code: Starting model inference Setup took: 0.0 seconds Resize took: 0.03741896152496338 seconds Concatenation took: 0.3359949588775635 seconds Normalization took: 0.9906361103057861 seconds Model prediction took: 0.3425499200820923 seconds Argmax took: 28.17007803916931 seconds Postprocess took: 0.054128050804138184 seconds Model inference took 29.934185028076172 seconds Here are the concatenateBuffers, normalizeBuffers, and argmax functions that I use: func concatenateBuffers(_ buffers: [CVPixelBuffer?]) -> CVPixelBuffer? { guard buffers.count == 3, let first = buffers[0] else { return nil } let width = CVPixelBufferGetWidth(first) let height = CVPixelBufferGetHeight(first) let targetChannels = 9 var concatenated: CVPixelBuffer? let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue] as CFDictionary CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, attrs, &concatenated) guard let output = concatenated else { return nil } CVPixelBufferLockBaseAddress(output, []) defer { CVPixelBufferUnlockBaseAddress(output, []) } guard let outputData = CVPixelBufferGetBaseAddress(output) else { return nil } let outputPtr = UnsafeMutablePointer<UInt8>(OpaquePointer(outputData)) // Lock all input buffers at once buffers.forEach { buffer in guard let buffer = buffer else { return } CVPixelBufferLockBaseAddress(buffer, .readOnly) } defer { buffers.forEach { CVPixelBufferUnlockBaseAddress($0!, .readOnly) } } // Process each input buffer for (frameIdx, buffer) in buffers.enumerated() { guard let buffer = buffer, let inputData = CVPixelBufferGetBaseAddress(buffer) else { continue } let inputPtr = UnsafePointer<UInt8>(OpaquePointer(inputData)) let bytesPerRow = CVPixelBufferGetBytesPerRow(buffer) let totalPixels = width * height // Process all pixels in one go for this frame for i in 0..<totalPixels { let y = i / width let x = i % width let inputOffset = y * bytesPerRow + x * 4 let outputOffset = i * targetChannels + frameIdx * 3 // BGR order to match numpy outputPtr[outputOffset] = inputPtr[inputOffset + 2] // B outputPtr[outputOffset + 1] = inputPtr[inputOffset + 1] // G outputPtr[outputOffset + 2] = inputPtr[inputOffset] // R } } return output } func normalizeBuffer(_ buffer: CVPixelBuffer?) -> MLMultiArray? { guard let input = buffer else { return nil } let width = CVPixelBufferGetWidth(input) let height = CVPixelBufferGetHeight(input) let channels = 9 CVPixelBufferLockBaseAddress(input, .readOnly) defer { CVPixelBufferUnlockBaseAddress(input, .readOnly) } guard let inputData = CVPixelBufferGetBaseAddress(input) else { return nil } let shape = [1, NSNumber(value: channels), NSNumber(value: height), NSNumber(value: width)] guard let output = try? MLMultiArray(shape: shape, dataType: .float32) else { return nil } let inputPtr = inputData.assumingMemoryBound(to: UInt8.self) let bytesPerRow = CVPixelBufferGetBytesPerRow(input) let ptr = UnsafeMutablePointer<Float>(OpaquePointer(output.dataPointer)) let totalSize = width * height for c in 0..<channels { for idx in 0..<totalSize { let h = idx / width let w = idx % width let inputIdx = h * bytesPerRow + w * channels + c ptr[c * totalSize + idx] = Float(inputPtr[inputIdx]) / 255.0 } } return output } func argmax(_ array: MLMultiArray) -> MLMultiArray? { let shape = array.shape.map { $0.intValue } guard shape.count == 3, shape[0] == 1, shape[1] == 256, shape[2] == 230400 else { return nil } guard let output = try? MLMultiArray(shape: [1, NSNumber(value: 230400)], dataType: .int32) else { return nil } let ptr = UnsafePointer<Float>(OpaquePointer(array.dataPointer)) let outputPtr = UnsafeMutablePointer<Int32>(OpaquePointer(output.dataPointer)) let channelSize = 230400 for pos in 0..<230400 { var maxValue = -Float.infinity var maxIndex: Int32 = 0 for channel in 0..<256 { let value = ptr[channel * channelSize + pos] if value > maxValue { maxValue = value maxIndex = Int32(channel) } } outputPtr[pos] = maxIndex } return output } Are there any glaring areas of inefficiencies that can be reduced to allow for under real time processing whilst following the same logic as found in the python code exactly? Would using Obj-C speed things up for some reason? Are there any tools I can use so I don't have to write these functions myself? Additionally, in the classes init, function, I tried to check the compute units being used since I feel 0.34 seconds for a singular model prediction is also far too long, but no print statements are showing for some reason: init() { guard let loadedModel = try? BallTrackerModel() else { fatalError("Could not load model") } let config = MLModelConfiguration() config.computeUnits = .all guard let configuredModel = try? BallTrackerModel(configuration: config) else { fatalError("Could not configure model") } self.model = configuredModel print("model loaded with compute units \(config.computeUnits.rawValue)") } Thanks!

I’m trying to use a Decimal as a @Property in my AppEntity, but using the following code shows me a compiler error. I’m using Xcode 16.1. The documentation notes the following: You can use the @Parameter property wrapper with common Swift and Foundation types: Primitives such as Bool, Int, Double, String, Duration, Date, Decimal, Measurement, and URL. Collections such as Array and Set. Make sure the collection’s elements are of a type that’s compatible with IntentParameter. Everything works fine for other primitives as bools, strings and integers. How do I use the Decimal though? Code struct MyEntity: AppEntity { var id: UUID @Property(title: "Amount") var amount: Decimal // … } Compiler Error This error appears at the line of the @Property definition: Generic class 'EntityProperty' requires that 'Decimal' conform to '_IntentValue'

基于iPhone 14 Max相机，实现模型识别，并在识别对象周围画一个矩形框。宽度和高度使用激光雷达计算，并在实时更新的图像上以厘米为单位显示。 swift code

Hi everyone😊, I want to implement facial recognition into my app. I was planning to use createML's image classification, but there seams to be a lot of hassle to implement (the JSON file etc.). Are there some other easy to implement options that don't involve advanced coding. Thanks, Oliver

I’m building an app that generates images based on text input from a specific text field. However, I’m encountering a problem: For short prompts like "a cat and a dog", the entire string is sent to the Image Playground, even when I use the extracted method. For longer inputs, the behavior is inconsistent. Sometimes it extracts keywords correctly, but other times it doesn’t extract anything at all. Since my app relies on generating images based on the extracted keywords, this inconsistency negatively impacts the user experience in my app. How can I make sure that keywords are always extracted from the input string? Button("Generate", systemImage: "apple.intelligence") { isPresented = true } .imagePlaygroundSheet(isPresented: $isPresented, concepts: [ImagePlaygroundConcept.extracted(from: text, title: textTitle)]) { url in imageURL = url }

I have exported a Pytorch model into a CoreML mlpackage file and imported the model file into my iOS project. The model is a Music Source Separation model - running prediction on audio-spectrogram blocks and returning separated audio source spectrograms. Model produces correct results vs. desktop+GPU+Python but the inference on iPhone 15 Pro Max is really, really slow. Using Xcode model Performance tool I can see that the inference isn't automatically managed between compute units - all of it runs on CPU. The Performance tool notation hints all that ops should be supported by both the GPU and Neural Engine. One thing to note, that when initializing the model with MLModelConfiguration option .cpuAndGPU or .cpuAndNeuralEngine there is an error in Xcode console: `Error(s) occurred compiling MIL to BNNS graph: [CreateBnnsGraphProgramFromMIL]: Failed to determine convolution kernel at location at /private/var/containers/Bundle/Application/2E3C4AFF-1FA4-4C95-AAE4-ECEBC0FB0BF9/mymss.app/mymss.mlmodelc/model.mil:2453:12 @ CreateBnnsGraphProgramFromMIL` Before going back hammering the model in Python, are there any tips/strategies I could try in CoreMLTools export phase or in configuring the model for prediction on iOS? My export toolchain is currently Linux with CoreMLTools v8.1, export target iOS16.

I'm working on my Swift Student Challenge submission and developing a Vision framework-based image classifier. I want to ensure I'm following best practices for training data and follow to guidelines for what images I use to train my image classifier. What types of images can I use for training my model? Are there specific image databases or resources recommended by Apple that are safe to use for Swift Student Challenge submissions? Currently considering images used from wikipedia, and my own images

FB:FB16079804 Hello, I've made the FastAI's Cat vs Dog model into model that distinguishes lemons from limes and it all works fine in a notebook. I am now looking to transform this model into Core ML for my iOS app using TorchScript and Apple official guidelines for coremltools. Model converts but I cannot see the Preview Tab in. Xcode. Have anyone of you tried to convert to Core ML? I guess my input types are not matching with coremltools expectations for preview but I am stuck . Here is my code. import torch import coremltools as ct from fastai.vision.all import * import json from torchvision import transforms # Load your Fastai model (replace with your actual path) learn = load_learner('lemonmodel.pkl') # Example input image (you can use any image from your dataset) input_image = PILImage.create('example.jpg') # Preprocess the image (assuming you used these transforms during training) to_tensor = transforms.ToTensor() normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) input_tensor = to_tensor(input_image) input_tensor = normalize(input_tensor) # Apply normalization # Add a batch dimension input_tensor = input_tensor.unsqueeze(0) # Ensure float32 type input_tensor = input_tensor.float() # Trace the model trace = torch.jit.trace(learn.model, input_tensor) # Define the Core ML input type (considering your model's input shape) _input = ct.ImageType( name="input_1", shape=input_tensor.shape, bias=[-0.485/0.229, -0.456/0.224, -0.406/0.225], scale=1./(255*0.226) ) # Convert the model to Core ML format mlmodel = ct.convert( trace, inputs=[_input], minimum_deployment_target=ct.target.iOS14 # Optional, set deployment target ) # Set model type as 'imageClassifier' for the Preview tab mlmodel.type = 'imageClassifier' # Correct structure for preview parameters** (assuming two classes: 'lemon' and 'lime') labels_json = { "imageClassifier": { "labels": ["lemon", "lime"], "input": { "shape": list(input_tensor.shape), # Provide the actual input shape "mean": [0.485, 0.456, 0.406], # Match normalization mean "std": [0.229, 0.224, 0.225] # Match normalization std }, "output": { "shape": [1, 2] # Output shape for your model (2 classes) } } } # Setting up the metadata with correct 'preview' params mlmodel.user_defined_metadata['com.apple.coreml.model.preview.params'] = json.dumps(labels_json) # Save the model as .mlmodel mlmodel.save("LemonClassifierGemini.mlmodel") mlmodel = ct.convert( trace, inputs=[_input], minimum_deployment_target=ct.target.iOS14 # Optional, set deployment target ) # Set model type as 'imageClassifier' for the Preview tab** mlmodel.type = 'imageClassifier' # Correct structure for preview parameters** (assuming two classes: 'lemon' and 'lime') labels_json = { "imageClassifier": { "labels": ["lemon", "lime"], "input": { "shape": list(input_tensor.shape), # Provide the actual input shape "mean": [0.485, 0.456, 0.406], # Match normalization mean "std": [0.229, 0.224, 0.225] # Match normalization std }, "output": { "shape": [1, 2] # Output shape for your model (2 classes) } } } # Setting up the metadata with correct 'preview' params** mlmodel.user_defined_metadata['com.apple.coreml.model.preview.params'] = json.dumps(labels_json) # Save the model as .mlmodel mlmodel.save("LemonClassifierGemini.mlmodel") My model is : Input batch shape: torch.Size([32, 3, 192, 192]) Labels batch shape: torch.Size([32]) Validation Loss: None, Validation Metric: None Predictions shape: torch.Size([63, 2]) Targets shape: torch.Size([63]) Code for the model : searches = 'lemon','lime' path = Path('lemon_or_not') for o in searches: dest = (path/o) dest.mkdir(exist_ok=True, parents=True) download_images(dest, urls=search_images(f'{o} photo')) time.sleep(5) resize_images(path/o, max_size=400, dest=path/o) dls = DataBlock( blocks=(ImageBlock, CategoryBlock), get_items=get_image_files, splitter=RandomSplitter(valid_pct=0.2, seed=42), get_y=parent_label, item_tfms=[Resize(192, method='squish')] ).dataloaders(path, bs=32) dls.show_batch(max_n=6) learn = vision_learner(dls, resnet18, metrics=error_rate) learn.fine_tune(3) is_lemon,_,probs = learn.predict(PILImage.create('lemon.jpg')) print(f"This is a: {is_lemon}.") print(f"Probability it's a lemon: {probs[0]:.4f}") This is a: lemon. Probability it's a lemon: 1.0000 learn.export('lemonmodel.pkl') I am stuck to why it doest show the Preview Tab.

I am a App designer and I am curious about what specific ML or AI Apple used to develop those features in the system. As far as I know, Apple's hand-raising detection, destination recommendations in maps, and exercise types in fitness all use ML. Are there more specific application examples of ML or AI? Does Apple have a document specifically introducing examples of specific applications of ML or AI technology in the system?

Hi all, I’m encountering an issue when trying to run Apple Foundation Models in a blank project targeting iOS 26. Below are the details: Xcode: Latest version with iOS 26 SDK macOS: macOS 26 Tahoe (installed on main disk) Mac: 16” MacBook Pro with M2 Pro chip Apple Intelligence: Available and functional on this machine Problem: I created a new blank iOS project, set the deployment target to iOS 26, and ran the following minimal code using Foundation Models. However, I get no response at all in the output - not even an error. The app runs, but the model does not produce any output. #Playground { let session = LanguageModelSession() let response = try await session.respond(to: "Tell me a story") } Then, I tried to catch an error with this code: #Playground { let session = LanguageModelSession() do { let response = try await session.respond(to: "Tell me a story") print(response) } catch { print("Failed to get response:", error) } print("This line, never gets executed") } And got these results: I’ve done further testing and discovered something important: I tried running the Code Along sample project, and there the #Playground macro worked without issues. The only significant difference I noticed was the Canvas run destination: In my original project, I was using iPhone 16 Pro (iOS 26) as the run target in Canvas. Apple Intelligence was enabled on the simulator, but no response was returned when executing the prompt. In the sample project, the Canvas was running on My Mac. I attempted to match that setup, but at first, my destination was My Mac (Designed for iPad), which still didn’t work. The macro finally executed properly once I switched to My Mac (AppKit). So the question is ... it seems that for now, Foundation Models and the #Playground macro only run correctly when the canvas or destination is set to “My Mac (AppKit)”?

While training a text classifier model with a few thousand samples completes in seconds, when using 100,000 or 1 million samples, CreateML's training time increases exponentially (to hours or days). During these hours/days, GPU usage is low and almost every CPU core is idle. When using the Swift APIs for model training, resource utilization does not increase. I'm using Xcode 16.2, macOS 15.2 on either an M2 Ultra 64 GB or an M3 Max 48 GB laptop (both using built-in SSD with ~500 GB free) running no other applications. Is there a setting I've missed to allow training to take over more of my computing resources? Is this expected of CreateML (i.e., when looking to exploit a larger corpus, I should move to other tooling)? I'd love to speed up my iteration cycle time.

Hello, I have a CoreML model and I want to convert it to a PyTorch model. Any ideas if this is possible and if so how?

Alguem me pode indicar se os devolopers que estão no espaço da união europeia, possam aceder aos serviços apple intelligence ? Obrigado

Do we know what a safe max token limit is? After some iterating, I have come to believe 4096 might be the limit on device. Could you help me out by answering any of these questions: Is 4096 the correct limit? Do all devices have the same limit? Will the limit change over time or by device? The errors I get when going over the limit do not seem to say, hey you are over, so it's just by trial and error that I figure these issues out. Thanks for the fun new toys. Regards, Rob

Hi, I am trying to create a multi label image classifier model using CreateML (the one included in Xcode 16.1). However, my annoations.json file won't get accepted by the app. I get the following error: annotations.json file contains field "Index 0" that is not of type String Here is a JSON example which results in said error: [ { "image": "image1.jpg", "annotations": [ { "label": "car-license-plate", "coordinates": { "x": 160, "y": 108, "width": 190, "height": 200 } } ] }, { "image": "image2.jpg", "annotations": [ { "label": "car-license-plate", "coordinates": { "x": 250, "y": 150, "width": 100, "height": 98 } } ] } ]

Is there anywhere we can reference error codes? I'm getting this error: "The operation couldn’t be completed. (FoundationModels.LanguageModelSession.GenerationError error 4.)" and I have no idea of what it means or what to attempt to fix.

Does CoreML object detection only support AABB (Axis-Aligned Bounding Boxes) or also OBB (Oriented Bounded Boxes)? If not, any way to do it using Apple frameworks?

Hey guys, I've been having difficulties transferring my Xcode project to a Swift playground (.swiftpm) for the Swift Student Challenge. I keep getting these errors as well as none of the views being able to find the model in scope: "TrashDetector 1.mlmodel: No predominant language detected. Set COREML_CODEGEN_LANGUAGE to preferred language." Unexpected duplicate tasks: Target 'TrashQuest' (project 'TrashQuest') has write command with output /Users/kmcph3/Library/Developer/Xcode/DerivedData/TrashQuest-glvzskunedgtakfrdmsxdoplondj/Build/Intermediates.noindex/TrashQuest.build/Debug-iphonesimulator/TrashQuest.build/0a4ef2429d66360920ddb4f16e65e233.sb I've gone through multiple post with these exact problems, but they all seem to be talking about ".playground" files due to the "Resources" folder (mind you I did try exactly what they said). Is there anyone that can help??? (Quick side note, why does it need to be a swiftpm file for the SSC??? Like why can't we just send the zip of our Xcode project??)

I have an issue with AI writing tools, certain applications, such as LinkedIn, function effectively, while others, like Instagram and WhatsApp, lack the writing tools option.