On accessing the CoreML Model Deployment dashboard with my developer account, the page gives a bad request error saying "Your request was invalid". Also, when I try to create a Model Collection it gives an error saying "One of the fields was invalid".

Hi, I have a core ml model that when I try to print the: modelPrediction?.labelProbability which is of type String:Double and contains all the features with their corresponding probabilities, the value of type double comes with nan rest = nan right = nan up = nan Sometimes restarting makes it work again. Sometimes it can take a lot of restarts to start working again. Even when deleting the app and installing again the same thing happens. Also tried changing the deployment version but didn't seem to fix it. Any help is appreciated.

We've 10 CoreML models in our app, each encrypted with a separate key generated in XCode. After opening and closing the app 6-7 times, the app crashes at model initialization with error: 2021-04-21 13:52:47.711729+0300 MyApp[95443:7341643] Fatal error: 'try!' expression unexpectedly raised an error: Error Domain=com.apple.CoreML Code=9 "Failed to generate key request for 08494FB2-B070-440F-A8A5-CBD0823A258E with error: -42905" UserInfo={NSLocalizedDescription=Failed to generate key request for 08494FB2-B070-440F-A8A5-CBD0823A258E with error: -42905}: file MyApp/Model.swift, line 43 Looks like iPhone is blocking the app for suspicious behavior and the app fails to decrypt the model. We noticed that after ~10 hours the app is unlocked, it successfully decrypts and initializes the model. Opening and closing the app many times in a short period of time is indeed unnatural, but the most important question is how to avoid blocking? Would Apple block the app if a user opens and closes it 10 times during a day? How does the number of models in the app affect probability that the app will be blocked? Thanks!

With the release of Xcode 13, a large section of my vision framework processing code became errors and cannot compile. All of these have became deprecated. This is my original code:  do {       // Perform VNDetectHumanHandPoseRequest       try handler.perform([handPoseRequest])       // Continue only when a hand was detected in the frame.       // Since we set the maximumHandCount property of the request to 1, there will be at most one observation.       guard let observation = handPoseRequest.results?.first else {         self.state = "no hand"         return       }       // Get points for thumb and index finger.       let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb)       let indexFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyIndexFinger)       let middleFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyMiddleFinger)       let ringFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyRingFinger)       let littleFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyLittleFinger)       let wristPoints = try observation.recognizedPoints(forGroupKey: .all)               // Look for tip points.       guard let thumbTipPoint = thumbPoints[.handLandmarkKeyThumbTIP],          let thumbIpPoint = thumbPoints[.handLandmarkKeyThumbIP],          let thumbMpPoint = thumbPoints[.handLandmarkKeyThumbMP],          let thumbCMCPoint = thumbPoints[.handLandmarkKeyThumbCMC] else {         self.state = "no tip"         return       }               guard let indexTipPoint = indexFingerPoints[.handLandmarkKeyIndexTIP],          let indexDipPoint = indexFingerPoints[.handLandmarkKeyIndexDIP],          let indexPipPoint = indexFingerPoints[.handLandmarkKeyIndexPIP],          let indexMcpPoint = indexFingerPoints[.handLandmarkKeyIndexMCP] else {         self.state = "no index"         return       }               guard let middleTipPoint = middleFingerPoints[.handLandmarkKeyMiddleTIP],          let middleDipPoint = middleFingerPoints[.handLandmarkKeyMiddleDIP],          let middlePipPoint = middleFingerPoints[.handLandmarkKeyMiddlePIP],          let middleMcpPoint = middleFingerPoints[.handLandmarkKeyMiddleMCP] else {         self.state = "no middle"         return       }               guard let ringTipPoint = ringFingerPoints[.handLandmarkKeyRingTIP],          let ringDipPoint = ringFingerPoints[.handLandmarkKeyRingDIP],          let ringPipPoint = ringFingerPoints[.handLandmarkKeyRingPIP],          let ringMcpPoint = ringFingerPoints[.handLandmarkKeyRingMCP] else {         self.state = "no ring"         return       }               guard let littleTipPoint = littleFingerPoints[.handLandmarkKeyLittleTIP],          let littleDipPoint = littleFingerPoints[.handLandmarkKeyLittleDIP],          let littlePipPoint = littleFingerPoints[.handLandmarkKeyLittlePIP],          let littleMcpPoint = littleFingerPoints[.handLandmarkKeyLittleMCP] else {         self.state = "no little"         return       }               guard let wristPoint = wristPoints[.handLandmarkKeyWrist] else {         self.state = "no wrist"         return       } ... } Now every line from thumbPoints onwards results in error, I have fixed the first part (not sure if it is correct or not as it cannot compile) to :         let thumbPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.thumb.rawValue)        let indexFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.indexFinger.rawValue)        let middleFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.middleFinger.rawValue)        let ringFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.ringFinger.rawValue)        let littleFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.littleFinger.rawValue)        let wristPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.littleFinger.rawValue) I tried many different things but just could not get the retrieving individual points to work. Can anyone help on fixing this?

I implement a custom pytorch layer on both CPU and GPU following [Hollemans amazing blog] (https://machinethink.net/blog/coreml-custom-layers ). The cpu version works good, but when i implemented this op on GPU it cannot activate "encode" function. Always run on CPU. I have checked the coremltools.convert() options with compute_units=coremltools.ComputeUnit.CPU_AND_GPU, but it still not work. This problem also mentioned in https://stackoverflow.com/questions/51019600/why-i-enabled-metal-api-but-my-coreml-custom-layer-still-run-on-cpu and https://developer.apple.com/forums/thread/695640. Any idea on help this would be grateful. System Information mac OS: 11.6.1 Big Sur xcode: 12.5.1 coremltools: 5.1.0 test device: iphone 11

Hello, I'm new using CoreML and I'm trying to do a test app with the models that already exist. I'm having next error at the moment to classifier the image: [coreml] Failed to get the home directory when checking model path. I would like to receive your help to solve this error. Thanks.

Im on the recent version of MacOs and I recently trained a Style Transfer model using CreateML. I used the preview tab of CreateML to preview my model with a video (as well as an image), however when I press the button to export or share the result from the neural network none are exported. The modal window appears but doesnt save after the progress bar shows up for the conversion I tried converting the CoreML model file into a CoreML package, however when I tried exporting the preview it crashed and switched tabs to the package information section. I've been having this issue with all three export buttons on the model preview section of both the CreateML application and Xcode. Is this happening to anyone else? Ive also tried using the coremltools package for Python to extract a preview, however documentation for Style Transfer networks doesnt exist for loading videos with that package. The style transfer network only takes an input of images, so its unclear where a video file can be loaded.

I have tried many times. When I change the file or re-create it, it shows 404 error { "code": 400, "message": "InvalidArgumentError: Unable to unzip MLArchive", "reason": "There was a problem with your request.", "detailedMessage": "InvalidArgumentError: Unable to unzip MLArchive", "requestUuid": "699afb97-8328-4a83-b186-851f797942aa" }

I am trying to train an image classification network in Keras with tensorflow-metal. The training freezes after the first 2-3 epochs if image augmentation layers are used (RandomFlip, RandomContrast, RandomBrightness) The system appears to use both GPU as well as CPU (as indicated by Activity Monitor). Also, warnings appear both in Jupyter and Terminal (see below). When the image augmentation layers are removed (i.e. we only rebuild the head and feed images from disk), CPU appears to be idle, no warnings appear, and training completes successfully. Versions: python 3.8, tensorflow-macos 2.11.0, tensorflow-metal 0.7.1 Sample code: img_augmentation = Sequential( [ layers.RandomFlip(), layers.RandomBrightness(factor=0.2), layers.RandomContrast(factor=0.2) ], name="img_augmentation", ) inputs = layers.Input(shape=(384, 384, 3)) x = img_augmentation(inputs) model = tf.keras.applications.EfficientNetV2S(include_top=False, input_tensor=x, weights='imagenet') model.trainable = False x = tf.keras.layers.GlobalAveragePooling2D(name="avg_pool")(model.output) x = tf.keras.layers.BatchNormalization()(x) top_dropout_rate = 0.2 x = tf.keras.layers.Dropout(top_dropout_rate, name="top_dropout")(x) outputs = tf.keras.layers.Dense(179, activation="softmax", name="pred")(x) newModel = Model(inputs=model.input, outputs=outputs, name="EfficientNet_DF20M_species") reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_accuracy', factor=0.9, patience=2, verbose=1, min_lr=0.000001) optimizer = tf.keras.optimizers.legacy.SGD(learning_rate=0.01, momentum=0.9) newModel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy']) history = newModel.fit(x=train_ds, validation_data=val_ds, epochs=30, verbose=2, callbacks=[reduce_lr]) During training with image augmentation, Jupyter prints the following warnings while training the first epoch: WARNING:tensorflow:Using a while_loop for converting Bitcast cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting Bitcast cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2 cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting Bitcast cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting Bitcast cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2 cause there is no registered converter for this op. WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter cause there is no registered converter for this op. ... During training with image augmentation, Terminal keeps spamming the following warning: 2023-02-21 23:13:38.958633: I metal_plugin/src/kernels/stateless_random_op.cc:282] Note the GPU implementation does not produce the same series as CPU implementation. 2023-02-21 23:13:38.958920: I metal_plugin/src/kernels/stateless_random_op.cc:282] Note the GPU implementation does not produce the same series as CPU implementation. 2023-02-21 23:13:38.959071: I metal_plugin/src/kernels/stateless_random_op.cc:282] Note the GPU implementation does not produce the same series as CPU implementation. 2023-02-21 23:13:38.959115: I metal_plugin/src/kernels/stateless_random_op.cc:282] Note the GPU implementation does not produce the same series as CPU implementation. 2023-02-21 23:13:38.959359: I metal_plugin/src/kernels/stateless_random_op.cc:282] Note the GPU implementation does not produce the same series as CPU implementation. ... Any suggestions?

I am working on the neural network classifier provided on the coremltools.readme.io in the updatable->neural network section(https://coremltools.readme.io/docs/updatable-neural-network-classifier-on-mnist-dataset). I am using the same code but I get an error saying that the coremltools.converters.keras.convert does not exist. But this I know can be coreml version issue. Right know I am using coremltools version 6.2. I converted this model to mlmodel with .convert only. It got converted successfully. But I face an error in the make_updatable function saying the loss layer must be softmax output. Even the coremlt package API reference there I found its because the layer name is softmaxND but it should be softmax. Now the problem is when I convert the model from Keras sequential model to coreml model. the layer name and type change. And the softmax changes to softmaxND. Does anyone faced this issue? if I execute this builder.inspect_layers(last=4) I get this output [Id: 32], Name: sequential/dense_1/Softmax (Type: softmaxND) Updatable: False Input blobs: ['sequential/dense_1/MatMul'] Output blobs: ['Identity'] [Id: 31], Name: sequential/dense_1/MatMul (Type: batchedMatmul) Updatable: False Input blobs: ['sequential/dense/Relu'] Output blobs: ['sequential/dense_1/MatMul'] [Id: 30], Name: sequential/dense/Relu (Type: activation) Updatable: False Input blobs: ['sequential/dense/MatMul'] Output blobs: ['sequential/dense/Relu'] In the make_updatable function when I execute builder.set_categorical_cross_entropy_loss(name='lossLayer', input='Identity') I get this error ValueError: Categorical Cross Entropy loss layer input (Identity) must be a softmax layer output.

In the ml-ane-transformers repo, there is a custom LayerNorm implementation for the Neural Engine-optimized shape of (B,C,1,S). The coremltools documentation makes it sound like the layer_norm MIL op would support this natively. In fact, the following code works on CPU: B,C,S = 1,768,512 g,b = 1, 0 @mb.program(input_specs=[mb.TensorSpec(shape=(B,C,1,S)),]) def ln_prog(x): gamma = (torch.ones((C,), dtype=torch.float32) * g).tolist() beta = (torch.ones((C), dtype=torch.float32) * b).tolist() return mb.layer_norm(x=x, axes=[1], gamma=gamma, beta=beta, name="y") However it fails when run on the Neural Engine, giving results that are scaled by an incorrect value. Should this work on the Neural Engine?

This issue has already been raised a few times in the coremltools repo (here, here, and here). I'm reposting here because this may be an issue in CoreML itself. In short, converting Huggingface's Bert implementation from PyTorch to CoreML results in significantly different model outputs. This test was originally posted in one of the linked issues: import numpy as np import torch from transformers import AutoTokenizer, AutoModel import coremltools as ct MODEL_NAME = "bert-base-uncased" sentences = ["This is a test."] tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) model = AutoModel.from_pretrained(MODEL_NAME, torchscript=True).eval() encoded_input = tokenizer(sentences, return_tensors='pt') traced_model = torch.jit.trace(model, tuple(encoded_input.values())) scripted_model = torch.jit.script(traced_model) model = ct.convert(scripted_model, source="pytorch", inputs=[ct.TensorType(name="input_ids", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32), ct.TensorType(name="token_type_ids", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32), ct.TensorType(name="attention_mask", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32)], convert_to="mlprogram", compute_units=ct.ComputeUnit.CPU_ONLY) with torch.no_grad(): pt_out = scripted_model(**encoded_input) cml_inputs = {k: v.to(torch.int32).numpy() for k, v in encoded_input.items()} pred_coreml = model.predict(cml_inputs) np.testing.assert_allclose(pt_out[0].detach().numpy(), pred_coreml["hidden_states"], atol=1e-5, rtol=1e-4) Running this shows that the model outputs are highly divergent: Max absolute difference: 7.901174 Max relative difference: 3424.6594 By contrast, running the same test with Huggingface's Distilbert implementation (distilbert-base-uncased) shows a much smaller difference in output: Max absolute difference: 0.00523943 Max relative difference: 45.603153 Again, I'm not totally sure that this is an issue in CoreML, but it would be great to be able to run Bert based models with CoreML!

Hi, I am trying to manipulate some data that I receive from a coreml model. To do this I am attempting to use MLShapedArray and MLShapedArraySlice. However, the shaped array slice does not behave as expected. Here is a small code snippet that initialize a 3D shaped array, prints it, then slices it and prints the slice: import CoreML var x = MLShapedArray<Float>(scalars: [0,1,2,3,4,5], shape: [1,3,2]) for i in 0..<x.shape[0] { for j in 0..<x.shape[1] { for k in 0..<x.shape[2] { print("[\(i),\(j),\(k)] = \(x[i,j,k].scalar ?? 0)") } } } var y = x[0...0,1..<2] print(y) for i in 0..<y.shape[0] { for j in 0..<y.shape[1] { for k in 0..<y.shape[2] { print("[\(i),\(j),\(k)] = \(y[i,j,k].scalar ?? 0)") } } } My expectation is that the slice will yield the second pair of numbers 2,3 but instead I get: [0,0,0] = 0.0 [0,0,1] = 1.0 [0,1,0] = 2.0 [0,1,1] = 3.0 [0,2,0] = 4.0 [0,2,1] = 5.0 MLShapedArraySlice<Float>(shape: [1, 1, 2], baseShapedArray: 0.0 1.0 2.0 3.0 4.0 5.0 , baseIndices: [], sliceRanges: [Range(0..<1), Range(1..<2), Range(0..<2)], originIndices: [0, 1, 0]) [0,0,0] = 0.0 [0,0,1] = 1.0 As you can see from the details of the printed slice - its shape is correct and it has the proper ranges displayed - yet when I subscript it it yields the same as the original shaped array. This is not what I expected - however I can't find any documentation or examples on how to use the slicing with ranges (or much on the slices at all) Am I using it wrong or is it just not working correctly?

Hi, I'd like to know if I have to set ITSAppUsesNonExemptEncryption key to true when using encrypted CoreML models as explained in the WWDC20-10152 video Use model deployment and security with Core ML Thank you. Gio

The 2-d image frame is extracted from a live/ pre-recorded video where the camera is placed behind one player so that the complete tennis court is visible in the frame. The court detection and ball detection have been done using CoreML and Vision APIs. Next step is to detect the trajectory and the bounce point of the ball to see if the ball is in/out of the court for scoring and analysis. I've used VNDetectTrajectoryRequest to draw the trajectory of the ball and used the detected court boundingBox as the ROI for trajectory detection.The problem is I am not able to remove the extra noise (coming from player movement in each frame) from the detection as the player is also in ROI. Next, How should I proceed with the ball bounce detection? private func detectTrajectories(_ controller: CameraViewController, _ buffer : CMSampleBuffer, _ orientation : CGImagePropertyOrientation) throws { let visionHandler = VNImageRequestHandler(cmSampleBuffer: buffer, orientation: orientation, options: [:]) let normalizedFrame = CGRect(x: 0, y: 0, width: 1, height: 1) DispatchQueue.main.async { // Get the frame of the rendered view. self.trajectoryView.frame = controller.viewRectForVisionRect(normalizedFrame) self.trajectoryView.roi = controller.viewRectForVisionRect(normalizedFrame) } //setup trajectory request setUpDetectTrajectoriesRequestWithMaxDimension() do { // Help manage the real-time use case to improve the precision versus delay tradeoff. detectTrajectoryRequest.targetFrameTime = .zero // The region of interest where the object is moving in the normalized image space. detectTrajectoryRequest.regionOfInterest = normalizedFrame try visionHandler.perform([detectTrajectoryRequest]) } catch { print("Failed to perform the trajectory request: \(error.localizedDescription)") return } } func setUpDetectTrajectoriesRequestWithMaxDimension() { detectTrajectoryRequest = VNDetectTrajectoriesRequest(frameAnalysisSpacing: .zero, trajectoryLength: trajectoryLength, completionHandler: completionHandler) // detectTrajectoryRequest. detectTrajectoryRequest .objectMinimumNormalizedRadius = 0.003 detectTrajectoryRequest.objectMaximumNormalizedRadius = 0.005 } private func completionHandler(request: VNRequest, error: Error?) { if let e = error { print(e) return } guard let observations = request.results as? [VNTrajectoryObservation] else { return } let relevantTrajectory = observations.filter { $0.confidence > trajectoryDetectionConfidence} if let trajectory = relevantTrajectory.first { DispatchQueue.main.async { print(trajectory.projectedPoints.count) self.trajectoryView.duration = trajectory.timeRange.duration.seconds self.trajectoryView.points = trajectory.detectedPoints self.trajectoryView.performTransition(.fadeIn, duration: 0.05) if !self.trajectoryView.fullTrajectory.isEmpty { self.trajectoryView.roi = CGRect(x: 0, y: 0, width: 1, height: 1) } } DispatchQueue.main.asyncAfter(deadline: .now() + 1.5, execute: { self.trajectoryView.resetPath() }) } } In the completion handler function, I have removed all the VNTrajectoryObservation that have a confidence of less than 0.9. After that, I have created a trajectoryView that displays the detected trajectory on the frame.

I have a csv file with values ranges between 0-1. When converted this csv values to MLMultiArray and then to UIImage it shows grey image. But the image is actually coloured. So is there any step missing or do I have to perform any more action to get this image to be coloured. Sample csv values - [0.556862745 0.62745098 0.811764706]

I am trying to use this repository https://github.com/Ma-Dan/Yolact-CoreML to test how fast the YOLACT Model would run on a new iPad, but I am getting this error: Thread 1: EXC_BAD_ACCESS (code=1, address=0x0) On this line: #0 0x00000001a2d19830 in Espresso::ANECompilerEngine::transpose_kernel::is_valid_for_engine(std::__1::shared_ptrEspresso::kernels_validation_status_t, Espresso::base_kernel::validate_for_engine_args_t const&) const () I modified line 37 on the Yolact.swift file that was throwing this error from: let model = yolact() to let model: yolact = { do { let config = MLModelConfiguration() return try yolact(configuration: config) } catch{ print(error) fatalError("Couldn't create model") } }() Removing the only code warning that I got. I don't know what else I can do.

Recently we added model encryption for an application we work on. While doing so, our tests notified us about changes in model inference results. I built a small application to reproduce the phenomena. While the changes are quite small (about 1e-9 maximum), we do want to know what causes the changes. We assumed encrypting a model does not alter it at all and the result should be bit-exact the same before and after. Anyone else experienced this? Is there an explanation why this changes happen?

I have a csv file with values ranges between 0-1. When converted this csv values to MLMultiArray and then to UIImage it shows grey image. But the image is actually coloured. So is there any step missing or do I have to perform any more action to get this image to be coloured. Sample csv values - [0.556862745 0.62745098 0.811764706] Code func parseCSV(data: String) -> [Float] { var finalArray = [Float]() var rows = data.components(separatedBy: "\n") for row in rows { let columns = row.components(separatedBy: ",") if columns.count == 3 { let r = columns[0] let g = columns[1] var b = columns[2] if b.contains("\r"){ b = b.replacingOccurrences(of: "\r", with: "") } finalArray.append(Float(r)!) finalArray.append(Float(g)!) finalArray.append(Float(b)!) } } return finalArray } let m = try MLMultiArray(shape: [1, 768, 512, 3], dataType: .double) for (index, element) in data.enumerated() { m[index] = NSNumber(value: element) } let model: ModelInput = { do { let config = MLModelConfiguration() return try ModelInput(configuration: config) } catch { fatalError("Couldn't create Prediction model") } }() let op = try model.prediction(input: ModelInput(input1: m)) let opvalue = op.featureValue(for: "Output") let multiArray = opvalue!.multiArrayValue!

Hey guys, I converted a T5-base (encoder/decoder) model to a CoreML model using https://github.com/huggingface/exporters (which are using coremltools under the hood). When creating a performance report for the decoder model within XCode it shows that all compute units are mapped to the CPU. This is also the experience I have when profiling the model (GPU and ANE are not used). I was under the impression that CoreML would divide up the layers and run those that can run on the GPU / ANE, but maybe I misunderstood. Is there anything I can do to get this to not run on the CPU exclusively?