My App has several resources that I'd like to spring open through App Intents. For example a series of Dictionaries. These resources however in the app are behind a log in (for security) and are entitlements that are purchased. They may own 4 of 7 dictionaries. If I want to have an intent that says, "Open Dictionary: (Dict Name)" how do I best handle situations where the user may no longer be logged in or have the entitlement for that specific dictionary? Thanks

I'm trying to convert a TensorFlow model that I didn't create and know approximately nothing about to CoreML so that I can use it in some functional tests. I can't tell you much about the model, but you can read about it on the blog from the team that created it: https://research.google/blog/improving-mobile-app-accessibility-with-icon-detection/ I can't convert this model to a TensorFlow Lite model because it uses a few full TensorFlow operations (which I could work around) and it exceeds the 4-tensor output limit (which I can't, AFAIK). So instead, I'm trying to convert the model to CoreML so that I can run it on-device. The issue I'm running into is that every approach fails in different ways. If I load the model with tf.saved_model.load and pass that as the first parameter to the convert call, it says NotImplementedError: Expected model format: [SavedModel | concrete_function | tf.keras.Model | .h5 | GraphDef], got <tensorflow.python.trackable.autotrackable.AutoTrackable object at 0x30d90c250> If I pass model.signatures['serving_default'] as the first parameter to convert, I get NotImplementedError: Expected model format: [SavedModel | concrete_function | tf.keras.Model | .h5 | GraphDef], got ConcreteFunction [...a page or two of info about the function here...] If I try to wrap it in a Keras layer using the instructions provided in the converter, it fails because a sequential model can't have multiple outputs. If I try to use a tf.keras.layers.TFSMLayer to load the model, it fails because there are multiple tags, and there's no way to specify tags when constructing the layer. (It tells me that I need to add 'tags' to load the model, but if I do that, it tells me that tags isn't a valid parameter to the call.) If I load the model with tf.saved_model.load and specify a single tag, then re-save it in a different location with tf.saved_model.save to generate a new model with only a single tag, then do input_layer = tf.keras.Input(shape=(768, 768, 3), dtype="int8") layer = tf.keras.layers.TFSMLayer("./serve_model", call_endpoint='serving_default') outputs = layer(input_layer) model = tf.keras.Model(input_layer, outputs) I get AttributeError: 'Functional' object has no attribute '_get_save_spec' At one point, I also tried this: class LayerFromSavedModel(tf.keras.layers.Layer): def __init__(self): super(LayerFromSavedModel, self).__init__() self.vars = legacy_model.variables def call(self, inputs): return legacy_model.signatures['serving_default'](inputs) input = tf.keras.Input(shape=(3000, 3000, 3)) model = tf.keras.Model(input, LayerFromSavedModel()(input)) and saw a similar failure. I've run out of ideas here. Is there simply no support whatsoever in the converter for importing a TensorFlow 2 SavedModel into CoreML, or am I missing something fundamental?

Hello. Where can I find some examples on creating custom genmojis in Swift and reusing it in an App?

Hellooo, I’m looking to implement an OpenAI assistant using APIs but I want to do this locally on a group of files. I want to be able to train a GPT on the contents of a folder for example. Does anyone have any experience in this? It seems OpenAI needs a lot of uploading on each request if I were to do this with their API after playing around (but this feels like I’m missing something). It’s also quite costly to use. I was hoping to use local machine learning and models but this is quite limited in what it can do (eg Lumachain)

Here is an App using CoreML API with ML package format, it works fine in iOS17, while it is crashed when calling [MLModel modelWithContentsOfURL ] to load model running in iOS18. It seems an exception is raised "Failed to set compute_device_types_mask E5RT: Cannot provide zero compute device types. (1)". Is it a bug of iOS18 beta version , and it will be fixed in the future? The stack is as below: Exception Codes: #0 at 0x1e9280254 Crashed Thread: 49 Application Specific Information: *** Terminating app due to uncaught exception 'NSGenericException', reason: 'Failed to set compute_device_types_mask E5RT: Cannot provide zero compute device types. (1)' Last Exception Backtrace: 0 CoreFoundation 0x0000000199466418 __exceptionPreprocess + 164 1 libobjc.A.dylib 0x00000001967cde88 objc_exception_throw + 76 2 CoreFoundation 0x0000000199560794 -[NSException initWithCoder:] 3 CoreML 0x00000001b4fcfa8c -[MLE5ProgramLibraryOnDeviceAOTCompilationImpl createProgramLibraryHandleWithRespecialization:error:] + 1584 4 CoreML 0x00000001b4fcf3cc -[MLE5ProgramLibrary _programLibraryHandleWithForceRespecialization:error:] + 96 5 CoreML 0x00000001b4fc23d8 __44-[MLE5ProgramLibrary prepareAndReturnError:]_block_invoke + 60 6 libdispatch.dylib 0x00000001a12e1160 _dispatch_client_callout + 20 7 libdispatch.dylib 0x00000001a12f07b8 _dispatch_lane_barrier_sync_invoke_and_complete + 56 8 CoreML 0x00000001b4fc3e98 -[MLE5ProgramLibrary prepareAndReturnError:] + 220 9 CoreML 0x00000001b4fc3bc0 -[MLE5Engine initWithContainer:configuration:error:] + 220 10 CoreML 0x00000001b4fc3888 +[MLE5Engine loadModelFromCompiledArchive:modelVersionInfo:compilerVersionInfo:configuration:error:] + 344 11 CoreML 0x00000001b4faf53c +[MLLoader _loadModelWithClass:fromArchive:modelVersionInfo:compilerVersionInfo:configuration:error:] + 364 12 CoreML 0x00000001b4faedd4 +[MLLoader _loadModelFromArchive:configuration:modelVersion:compilerVersion:loaderEvent:useUpdatableModelLoaders:loadingClasses:error:] + 540 13 CoreML 0x00000001b4f9b900 +[MLLoader _loadWithModelLoaderFromArchive:configuration:loaderEvent:useUpdatableModelLoaders:error:] + 424 14 CoreML 0x00000001b4faaeac +[MLLoader _loadModelFromArchive:configuration:loaderEvent:useUpdatableModelLoaders:error:] + 460 15 CoreML 0x00000001b4fb0428 +[MLLoader _loadModelFromAssetAtURL:configuration:loaderEvent:error:] + 240 16 CoreML 0x00000001b4fb00c4 +[MLLoader loadModelFromAssetAtURL:configuration:error:] + 104 17 CoreML 0x00000001b5314118 -[MLModelAssetResourceFactoryOnDiskImpl modelWithConfiguration:error:] + 116 18 CoreML 0x00000001b5418cc0 __60-[MLModelAssetResourceFactory modelWithConfiguration:error:]_block_invoke + 72 19 libdispatch.dylib 0x00000001a12e1160 _dispatch_client_callout + 20 20 libdispatch.dylib 0x00000001a12f07b8 _dispatch_lane_barrier_sync_invoke_and_complete + 56 21 CoreML 0x00000001b5418b94 -[MLModelAssetResourceFactory modelWithConfiguration:error:] + 276 22 CoreML 0x00000001b542919c -[MLModelAssetModelVendor modelWithConfiguration:error:] + 152 23 CoreML 0x00000001b5380ce4 -[MLModelAsset modelWithConfiguration:error:] + 112 24 CoreML 0x00000001b4fb0b3c +[MLModel modelWithContentsOfURL:configuration:error:] + 168

Hei Sir is not working in ios beta 18

I consistently receive corrupted results from tf.signal.fft3d() when it is within a function that has a @tf.function decorator. The results are all zero (0.) for entries after a certain x index (see image). Surprisingly, the issue depends on the matrix size. For example, (1023, 1023, 287) works but (1023, 1023, 575) does not. The issue is problematic because it occurs silently and not for all matrix sizes, i.e. can easily slip through tests. The error occurs only when tensorflow-metal is installed. The Tensorflow version is 2.16.1. My hardware is a Macbook Pro M3 Max with 40 GPU cores, 128 GB RAM running MacOS Sonoma version 14.5 (23F79). A Python environment to reproduce the bug can be created as follows: conda create --name tfmetalbug python=3.11.9 conda activate tfmetalbug pip install tensorflow tensorflow-metal conda install matplotlib The following code reproduces the issue: import tensorflow as tf import numpy as np import matplotlib.pyplot as plt # Wrap fft3d with tf.function @tf.function def fft3d_wrapper_function(x): return tf.signal.fft3d(x) # Generate a 3D image img = tf.random.normal(shape=(1023, 1023, 575), stddev=1., dtype=float) # generate random 3d image img = tf.dtypes.cast(img, tf.complex64) # convert to complex values # Compute the 3D FFT img_fft = fft3d_wrapper_function(img) # Visualize the 3D FFT plt.imshow(np.real(img_fft)[:, img_fft.shape[1]//2+10, :], cmap="gray", vmin=-0.001, vmax=0.001) plt.savefig("fft3d_wrapper_function.png") For me, removing the @tf.function decorator has resolved the issue.

I am testing the new scaled dot product attention CoreML op on macOS 15 beta 1. Based on the session video I was expecting to see a speedup when running on GPU however I see roughly equivalent performance to the same model on macOS 14. I ran tests with two models: one that simply repeats y = sdpa(y, k, v) 50 times gpt2 124M converted from nanoGPT (the only change is not returning loss from the forward method) I converted both models using coremltools 8.0b1 with minimum deployment targets of macOS 14 and also macOS 15. In Xcode, I can see that the new op was used for the macOS 15 target. Running on macOS 15 both target models take the same time, and that time matches the runtime on macOS 14. Should I be seeing performance improvements?

When in Safari, you can say something like, "Siri, text this link to mom" or "Siri, save this link to reminders" and it will do it with the currently viewed link. Shortcuts also has a "Get what's on screen" action that can be added. How do I expose the user's current context to my App Intent?

The DINO v1/v2 models are particularly interesting to me as they produce embeddings for the detected objects rather than ordinary classification indexes. That makes them so much more useful than the CNN based models. I would like to prepare some of the models posted on Huggingface to run on Apple Silicon, but it seems that the default conversion with TorchScript will not work. The other default conversions I've looked at so far also don't work. Conversion based on an example input doesn't capture enough of the model. I know that some have managed to convert it as I have a demo with a coreml model that seems to work, but I would like to know how to do the conversion myself. Has anyone managed to convert any of the DINOv2 models?

I'm playing with the new Vision API for iOS18, specifically with the new CalculateImageAestheticsScoresRequest API. When I try to perform the image observation request I get this error: internalError("Error Domain=NSOSStatusErrorDomain Code=-1 \"Failed to create espresso context.\" UserInfo={NSLocalizedDescription=Failed to create espresso context.}") The code is pretty straightforward: if let image = image { let request = CalculateImageAestheticsScoresRequest() Task { do { let cgImg = image.cgImage! let observations = try await request.perform(on: cgImg) let description = observations.description let score = observations.overallScore print(description) print(score) } catch { print(error) } } } I'm running it on a M2 using the simulator. Is it a bug? What's wrong?

I am very new to App Intents and I am trying to add them to my On Device LLM ChatBot app so my users can get answers to any questions anywhere in iOS. I have the following code and it is working wonderfully in the Shortcuts app. import AppIntents struct AskAi: AppIntent { static var openAppWhenRun: Bool = false static let title: LocalizedStringResource = "Ask Ai About" static let description = "Gets an answer from Ai for your question." @Parameter(title: "Question") var question: String static var parameterSummary: some ParameterSummary { Summary("Ask Ai About \(\.$question)") } @MainActor func perform() async throws -> some IntentResult & ReturnsValue<String> { let bot: Bot = Bot() await bot.respond(to: self.question) return .result( value: bot.output ) } } class AppShortcuts: AppShortcutsProvider { static var appShortcuts: [AppShortcut] { AppShortcut( intent: AskAi(), phrases: [ "Ask \(.applicationName) \(\.$question)", "Get \(.applicationName) answer for \(\.$question)", "Open \(\.$question) using \(.applicationName) ", "Using \(.applicationName) get help with \(\.$question)" ], shortTitle: "Ask Ai", systemImageName: "sparkles" ) } } I can create a shortcut for this AppIntent and that allows me say speak the response. I can call my shortcut via iOS 18 Beta 1 by the Shortcut name I set in the Shortcuts app and that allows it to work. It does not work at all by just Asking Siri any of the phrases I have defined. The info.plist has an app name alias defined just to be sure. I even added the Siri capability in Xcode-beta. I also tried using the ProvidesDialog return type too. Whatever I do the AppIntent is invisible to Siri. Siri tries to search the web, looking for my app name in the contacts or have an error Apple Cash which has nothing to do with what I was talking about. Is there anything else I am missing for setting up iOS AppIntents to work with Siri?

Hi, I'd like to use a MLX model already in the MLX Community in my App. I understand I first need to convert it to Core ML format. Is there an easy way to do that considering MLX is an Apple project? It would be good if it was easier then I'd be more motivated to use MLX to train my models. Thanks Richard

In the WWDC24 What’s New In Create ML at 6:03 the presenter introduced TimeSeriesClassifier as a new component of Create ML Components. Where are documentation and code examples for this feature? My app captures accelerometer time series data that I want to classify. Thank you so much!

Using App Shortcuts with app intents, Siri only responds to the first shortcut defined in the app shortcut below. struct MementoShortcuts: AppShortcutsProvider { u/AppShortcutsBuilder static var appShortcuts: [AppShortcut] { AppShortcut( intent: SaveLinkIntent(), phrases: ["Add a link to \(.applicationName)", "Add \(\.$url) to \(.applicationName)", "Make a new link in \(.applicationName)", "Create a new link in \(.applicationName) from \(\.$url)"], shortTitle: "Add Link", systemImageName: "link.badge.plus" ) AppShortcut( intent: LinkViewedIntent(), phrases: [ "Mark a link I saved in \(.applicationName) as viewed", "Mark \(\.$link) as viewed in \(.applicationName)", "Set link in \(.applicationName) to viewed", "Change status of \(\.$link) to viewed in \(.applicationName)", ], shortTitle: "Mark Link as Viewed", systemImageName: "book" ) } } I have tried switching the order and she always uses the one that comes first. Both show up in the shortcuts app as an app shortcut, but only one shortcut is recognized by Siri even if I say the other one's phrase.

I have a model that uses ‘flatten’, and when I converted it to a Core ML model and ran it on Xcode with an iPhone XR, I noticed that ‘flatten’ was automatically converted to ‘reshape’. However, the NPU does not support ‘reshape’. howerver, I got the Resnet50 model on apple models and performance it on XCode with the same iphone XR, I can see the 'flatten' operator which run on NPU. On the other hand, when I used the following code to convert ResNet50 in PyTorch and ran it on Xcode Performance, the ‘flatten’ operation was converted to ‘reshape’, which then ran on the CPU. ? So I dont know how to keep 'flatten' operator when convert to ml model ? coreml tool 7.1 iphone XR ios 17.5.1 from torchvision import models import coremltools as ct import torch import torch.nn as nn network_name = "my_resnet50" torch_model = models.resnet50(pretrained=True) torch_model.eval() width = 224 height = 224 example_input = torch.rand(1, 3, height, width) traced_model = torch.jit.trace(torch_model, (example_input)) model = ct.convert( traced_model, convert_to = "neuralnetwork", inputs=[ ct.TensorType( name = "data", shape = example_input.shape, dtype = np.float32 ) ], outputs = [ ct.TensorType( name = "output", dtype = np.float32 ) ], compute_units = ct.ComputeUnit.CPU_AND_NE, minimum_deployment_target = ct.target.iOS14, ) model.save("my_resnet.mlmodel") ResNet50 on Resnet50.mlmodel My Convertion of ResNet50

Error when trying to generate CoreML performance report, message says The data couldn't be written because it isn't in the correct format. Here is the code to replicate the issue import numpy as np import coremltools as ct from coremltools.converters.mil import Builder as mb import coremltools.converters.mil as mil w = np.random.normal(size=(256, 128, 1)) wemb = np.random.normal(size=(1, 32000, 128)) # .astype(np.float16) rope_emb = np.random.normal(size=(1, 2048, 128)) shapes = [(1, seqlen) for seqlen in (32, 64)] enum_shape = mil.input_types.EnumeratedShapes(shapes=shapes) fixed_shape = (1, 128) max_length = 2048 dtype = np.float32 @mb.program( input_specs=[ # mb.TensorSpec(enum_shape.symbolic_shape, dtype=mil.input_types.types.int32), mb.TensorSpec(enum_shape.symbolic_shape, dtype=mil.input_types.types.int32), ], opset_version=mil.builder.AvailableTarget.iOS17, ) def flex_like(input_ids): indices = mb.fill_like(ref_tensor=input_ids, value=np.array(1, dtype=np.int32)) causal_mask = np.expand_dims( np.triu(np.full((max_length, max_length), -np.inf, dtype=dtype), 1), axis=0, ) mask = mb.gather( x=causal_mask, indices=indices, axis=2, batch_dims=1, name="mask_gather_0", ) # mask = mb.gather( # x=mask, indices=indices, axis=1, batch_dims=1, name="mask_gather_1" # ) rope = mb.gather(x=rope_emb.astype(dtype), indices=indices, axis=1, batch_dims=1, name="rope") hidden_states = mb.gather(x=wemb.astype(dtype), indices=input_ids, axis=1, batch_dims=1, name="embedding") return ( hidden_states, mask, rope, ) cml_flex_like = ct.convert( flex_like, compute_units=ct.ComputeUnit.ALL, compute_precision=ct.precision.FLOAT32, minimum_deployment_target=ct.target.iOS17, inputs=[ ct.TensorType(name="input_ids", shape=enum_shape), ], ) cml_flex_like.save("flex_like_32") If I remove the hidden states from the return it does work, and it also works if I keep the hidden states, but remove both mask, and rope, i.e, the report is generated for both programs with either these returns: return ( # hidden_states, mask, rope, ) and return ( hidden_states, # mask, # rope, ) It also works if I use a static shape instead of an EnumeratedShape I'm using macOS 15.0 and Xcode 16.0 Edit 1: Forgot to mention that although the performance report fails, the model is still able to make predictions

Is it possible to change the folder where .blob are stocked when we training a model ? I trying to train a model to track an object, and I'm limited to the extraordinary 256Gb of my 2024 Mac ! :(

For some reason YDF does not work with the ARM processor. An issue with mutex and destruction.

iOS 18 App Intents while supporting iOS 17 Hello, I have an existing app that supports iOS 17. I already have three App Intents but would like to add some of the new iOS 18 app intents like ShowInAppSearchResultsIntent. However, I am having a hard time using #available or @available to limit this ShowInAppSearchResultsIntent to iOS 18 only while still supporting iOS 17. Obviously, the ShowInAppSearchResultsIntent needs to use @AssistantIntent which is iOS 18 only, so I mark that struct as @available(iOS 18, *). That works as expected. It is when I need to add this "SearchSnippetIntent" intent to the AppShortcutsProvider, that I begin to have trouble doing. See code below: struct SnippetsShortcutsAppShortcutsProvider: AppShortcutsProvider { @AppShortcutsBuilder static var appShortcuts: [AppShortcut] { //iOS 17+ AppShortcut(intent: SnippetsNewSnippetShortcutsAppIntent(), phrases: [ "Create a New Snippet in \(.applicationName) Studio", ], shortTitle: "New Snippet", systemImageName: "rectangle.fill.on.rectangle.angled.fill") AppShortcut(intent: SnippetsNewLanguageShortcutsAppIntent(), phrases: [ "Create a New Language in \(.applicationName) Studio", ], shortTitle: "New Language", systemImageName: "curlybraces") AppShortcut(intent: SnippetsNewTagShortcutsAppIntent(), phrases: [ "Create a New Tag in \(.applicationName) Studio", ], shortTitle: "New Tag", systemImageName: "tag.fill") //iOS 18 Only AppShortcut(intent: SearchSnippetIntent(), phrases: [ "Search \(.applicationName) Studio", "Search \(.applicationName)" ], shortTitle: "Search", systemImageName: "magnifyingglass") } let shortcutTileColor: ShortcutTileColor = .blue } The iOS 18 Only AppShortcut shows the following error but none of the options seem to work. Maybe I am going about it the wrong way. 'SearchSnippetIntent' is only available in iOS 18 or newer Add 'if #available' version check Add @available attribute to enclosing static property Add @available attribute to enclosing struct Thanks in advance for your help.