Post not yet marked as solved
Most examples, including within documentation, of using CoreML with iOS involve the creation of the Model under Xcode on a Mac and then inclusion of the Xcode generated MLFeatureProvider class into the iOS app and (re)compiling the app. However, it’s also possible to download an uncompiled model directly into an iOS app and then compile it (background tasks) - but there’s no MLFeatureProvider class. The same applies when using CreateML in an iOS app (iOS 15 beta) - there’s no automatically generated MLFeatureProvider. So how do you get one? I’ve seen a few queries on here and elsewhere related to this problem, but couldn’t find any clear examples of a solution. So after some experimentation, here’s my take on how to go about it:
Firstly, if you don’t know what features the Model uses, print the model description e.g. print("Model: ",mlModel!.modelDescription). Which gives Model:
inputs: (
"course : String",
"lapDistance : Double",
"cumTime : Double",
"distance : Double",
"lapNumber : Double",
"cumDistance : Double",
"lapTime : Double"
)
outputs: (
"duration : Double"
)
predictedFeatureName: duration
............
A prediction is created by guard **let durationOutput = try? mlModel!.prediction(from: runFeatures) ** ……
where runFeatures is an instance of a class that provides a set of feature names and the value of each feature to be used in making a prediction. So, for my model that predicts run duration from course, lap number, lap time etc the RunFeatures class is:
class RunFeatures : MLFeatureProvider {
var featureNames: Set = ["course","distance","lapNumber","lapDistance","cumDistance","lapTime","cumTime","duration"]
var course : String = "n/a"
var distance : Double = -0.0
var lapNumber : Double = -0.0
var lapDistance : Double = -0.0
var cumDistance : Double = -0.0
var lapTime : Double = -0.0
var cumTime : Double = -0.0
func featureValue(for featureName: String) -> MLFeatureValue? {
switch featureName {
case "distance":
return MLFeatureValue(double: distance)
case "lapNumber":
return MLFeatureValue(double: lapNumber)
case "lapDistance":
return MLFeatureValue(double: lapDistance)
case "cumDistance":
return MLFeatureValue(double: cumDistance)
case "lapTime":
return MLFeatureValue(double: lapTime)
case "cumTime":
return MLFeatureValue(double: cumTime)
case "course":
return MLFeatureValue(string: course)
default:
return MLFeatureValue(double: -0.0)
}
}
}
Then in my DataModel, prior to prediction, I create an instance of RunFeatures with the input values on which I want to base the prediction:
var runFeatures = RunFeatures()
runFeatures.distance = 3566.0
runFeatures.lapNumber = 1.0
runFeatures.lapDistance = 1001.0
runFeatures.lapTime = 468.0
runFeatures.cumTime = 468.0
runFeatures.cumDistance = 1001.0
runFeatures.course = "Wishing Well Loop"
NOTE there’s no need to provide the output feature (“duration”) here, nor in the featureValue method above but it is required in featureNames.
Then get the prediction with guard let durationOutput = try? mlModel!.prediction(from: runFeatures)
Regards,
Michaela
Post not yet marked as solved
Does anyone know how to list several target variables in a regression model? Create ML seems to allow only one target variable.
Is there a different library (other than Create ML) I can import in swift to gain access to multi-target regression model building?
Post not yet marked as solved
I would like to know if there are some best practices for integrating Core ML models into a Core Image pipeline, especially when it comes to support for tiling.
We are using a CIImageProcessorKernel for integrating an MLModel-based filtering step into our filter chain. The wrapping CIFilter that actually calls the kernel handles the scaling of the input image to the size the model input requires.
In the roi(forInput:arguments:outputRect:) method the kernel signals that it always requires the full extent of the input image in order to produce an output (since MLModels don't support tiling).
In the process(with:arguments:output:) method, the kernel is performing the prediction of the model on the input pixel buffer and then copies the result into the output buffer.
This works well until the filter chain is getting more and more complex and input images become larger. At this point, Core Image wants to perform tiling to stay within the memory limits. It can't tile the input image of the kernel since we defined the ROI to be the whole image.
However, it is still calling the process(…) method multiple times, each time demanding a different tile/region of the output to be rendered. But since the model doesn't support producing only a part of the output, we effectively have to process the whole input image again for each output tile that should be produced.
We already tried caching the result of the model run between consecutive calls to process(…). However, we are unable to identify that the next call still belongs to the same rendering call, but for a different tile, instead of being a different rendering entirely, potentially with a different input image.
If we'd have access to the digest that Core Image computes for an image during processing, we would be able to detect if the input changed between calls to process(…). But this is not part of the CIImageProcessorInput.
What is the best practice here to avoid needless reevaluation of the model? How does Apple handle that in their ML-based filters like CIPersonSegmentation?
Post not yet marked as solved
Would it be possible to use the camera to capture the speed of an object moving moving(100+ mph)? at close range?
Post not yet marked as solved
Hi,
I'm experimenting with the new MLModelCollection stuff in Xcode 12 and iOS 14 and I'm having trouble.
I've created a Model Collection named "Collection" using the CoreML Model Deployment dashboard, added a Model to the collection, and created a deployment with an uploaded .mlarchive file. In the dashboard I can navigate to the deployment and see Model Assets, and I can download the assets (it appears to be a zip file that when opened contains weights and such). The mlmodel that I created the .mlarchive from works great when I drag and drop it into my project.
In my iOS app, following the instructions in the MLModelCollection help files, I call:
MLModelCollection.beginAccessing(identifier: "iNatTestCollectionID", completionHandler: modelCollectionAvailable)
My modelCollectionAvailable method looks like:
func modelCollectionAvailable(result: Result<MLModelCollection, Error>) {
switch result {
case .success(let collection):
print("Model collection `\(collection.identifier)` is available.")
print("Model deployment id is \(collection.deploymentID)")
print("Model collection entries are \(collection.entries)")
case .failure(let error):
print("got \(error) when accessing collection")
}
}
Unfortunately, the collection I get in the callback always has an empty deployment id and there are no entries in the collection.
I've tried waiting a few days in case there were syncing or deployment issues, but still nothing.
I do see this error in the console:
2020-11-03 15:12:27.600165-0800 ModelDeploymentTest[15009:910661] [coreml] MLModelCollection: experimentId unexpectedly not returned
But I'm not sure what that means.
Anyone have any advice?
Thanks,
alex
Post not yet marked as solved
When importing a CSV file with ~ 50 columns and ~200 rows the "MLDataTable(contentsOf: inputDataPath, options: parsingOptions)" command has issues parsing. Much of the data has "0" in the fields but sporadically there are decimal values. If I have a column where I have say 180 "0"s and the last 20 columns have decimal values the column identifies as an "Int" and lines are dropped during the parsing process.
Is there a way provide Column Type Hints?
Is there a way to force a column type?
Is MLDataTable only looking at a handful of rows and determining the column type?
Post not yet marked as solved
Hello,
is it possible to calculate the distance from iPhone to objects which are recognized by ML models? If yes, how to do so? I suppose using AVDepthData is not suitable for measuring distances in real-time.
Perhaps you know projects doing so. Thanks in advance!
Post not yet marked as solved
My activity classifier is used in tennis sessions, where there are necessarily multiple people on the court. There is also a decent chance other courts' players will be in the shot, depending on the angle and lens.
For my training data, would it be best to crop out adjacent courts?
Post not yet marked as solved
Running mlmodel's prediction on ANE, the leakyReLU layer causes large different output feature maps than running on CpuAndGpu and CPUOnly. And the results are totally wrong when using ANE, but results are right using gpu or cpu.
If I just replace the leakyReLU with ReLU in the mlmodel, the output feature maps only have small difference and outputs are all correct.
This problem occurs on iPhone with A12 and A13 chip, no matter the ios system is ios12, ios13, or ios14. But on A14 device, the problem is gone.
Is it a bug in coreml framework on A12/A13 devices?
Post not yet marked as solved
Hi,
I found that the Core ML on iOS 15 consumes almost 450MB larger memory than iOS 14 for prediction of my model.
It may easily cause running out of memory crash on small memory devices.
To reproduce the issue, I created a sample code as bellow. It shows around 450MB on iOS 15 but only about 4MB on iOS 14.X.
I have already submitted the issue through Feedback Assistance.
But does anyone have any idea to work around the issue?
I've confirmed the the issue still exists on iOS 15.1 Beta 2.
Any advice would be appreciated.
import CoreML
class ViewController: UIViewController {
@IBOutlet weak var memoryLabel: UILabel!
@IBAction func predAction(_ sender: Any) {
let modelConf = MLModelConfiguration()
modelConf.computeUnits = .cpuOnly
do {
let input = UnsafeMutablePointer<Float>.allocate(capacity: 512 * 1024 * 2)
input.initialize(repeating: 0.0, count: 512 * 1024 * 2)
let inputMultiArray = try MLMultiArray(dataPointer: input, shape: [1,512,1024,2], dataType: .float32, strides: [(512 * 1024 * 2) as NSNumber ,(1024 * 2) as NSNumber,2,1], deallocator: nil)
let model = try TestModel(configuration: modelConf)
let memBefore = os_proc_available_memory()
let _ = try model.prediction(input:TestModelInput(strided_slice_3_0: inputMultiArray))
let memAfter = os_proc_available_memory()
print("memory size for prediction",memBefore - memAfter)
memoryLabel.text = String(describing:(memBefore - memAfter) )
}
catch {
print("error")
}
}
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view.
}
}
Post not yet marked as solved
For a Create ML activity classifier, I’m classifying “playing” tennis (the points or rallies) and a second class “not playing” to be the negative class. I’m not sure what to specify for the action duration parameter given how variable a tennis point or rally can be, but I went with 10 seconds since it seems like the average duration for both the “playing” and “not playing” labels.
When choosing this parameter however, I’m wondering if it affects performance, both speed of video processing and accuracy. Would the Vision framework return more results with smaller action durations?
Post not yet marked as solved
I need to build a model to add to my app and tried following the Apple docs here.
No luck because I get an error that is discussed on this thread on the forum. I'm still not clear on why the error is occurring and can't resolve it.
I wonder if CreateML inside Playgrounds is still supported at all? I tried using the CreateML app that you can access through developer tools but it just crashes my Mac (2017 MBP - is it just too much of a brick to use for ML at this point? I should think not because I've recently built and trained relatively simple models using Tensorflow. + Python on this machine, and the classifier I'm trying to make now is really simple and doesn't have a huge dataset).
Post not yet marked as solved
Since there are only 80 class labels for existing YOLOv3 Coreml model, I want to add some more categories to be used in my app, can I do that? If this is possible, how can I do?
Post not yet marked as solved
Hello.I have the follow problem:I have the converted from Caffe(with deconvolutional layer) model which work correct if I run it with python and coremltools. But if I try to load it on Iphone the loader return assertation error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-82.1/MPSNeuralNetwork/Filters/MPSCNNConvolutionTranspose.mm:207: failed assertion `number of groups must be > 0'I now that nGroups correct (=2)Caffe section:layer {
name: "upscore"
type: "Deconvolution"
bottom: "conv10_new"
top: "upscore"
param {
lr_mult: 0
decay_mult: 0
}
convolution_param {
num_output: 2
bias_term: false
pad: 16
kernel_size: 64
group: 2
stride: 32
weight_filler {
type: "bilinear"
}
}
}Mlmodel spec section: layers {
name: "upscore"
input: "conv10_new"
output: "upscore"
convolution {
outputChannels: 2
kernelChannels: 1
nGroups: 2
kernelSize: 64
kernelSize: 64
stride: 32
stride: 32
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
startEdgeSize: 16
endEdgeSize: 16
}
borderAmounts {
startEdgeSize: 16
endEdgeSize: 16
}
}
}
isDeconvolution: true
weights {Any ideas?Thanks!PS. coremltools convert Caffe deconvolution layer not totally correct, it needed to swap outputChannels and kernelChannels (reshape weights in caffe model). For example in this sample caffe model has weights shape (2, 1, 64, 64) but coremltools expect (1, 2, 64, 64)
Post not yet marked as solved
How should I think about video quality (if it's important) when gathering training videos? Does higher video quality of training data make for better predictions, or should it more closely match the common use case (1080p I suppose, thinking about iPhones broadly)?
Post not yet marked as solved
While trying to learn about coreML, I ran into an issue. Using a model from Apple's website (MobileNetV2) I had no issues. When I tried to use my own model that I created, I ran into the issue and the localized description was "Could not create inference context" when using the iPhone simulator. After a quick search I tested this using the arm64 simulator and it worked just fine. I believe this is an m1 related bug because another forum said it worked without any issues on intel Mac, but not their m1.
if let data = self.animal.imageData {
do {
let modelFile = try! DogorCatmodel1(configuration: MLModelConfiguration())
let model = try VNCoreMLModel(for: modelFile.model)
let handler = VNImageRequestHandler(data: data)
let request = VNCoreMLRequest(model: model) { (request, error) in
guard let results = request.results as? [VNClassificationObservation] else {
print("Could not classify")
return
}
for classification in results {
var identifier = classification.identifier
identifier = identifier.prefix(1).capitalized + identifier.dropFirst()
print(identifier)
print(classification.confidence)
}
}
do {
try handler.perform([request])
}
catch {
print(error.localizedDescription)
print("Invalid")
}
}
catch {
print(error.localizedDescription)
}
}
}
Post not yet marked as solved
I'm trying to implement a 5-D input tensor version of custom grid_sample based on this work. For encoding one of the input tensor, grid to a MTLTexture as shown in officical guide, i need to transpose grid from (N×D_grid×W_grid×H_grid×3) to (N×3×D_grid×W_grid ×H_grid) by builder.add_transpose similar like shown here. But in my implementation, i find that adding this transpose op will make the custom layer always run on CPU. Without this layer, the data can get in GPU. I doubt will builder.add_transpose have an effect on this?
System Information
xcode: 13.2
coremltools: 4.1/5.1.0
test device: iphone 11
Post not yet marked as solved
Below, the sampleBufferProcessor closure is where the Vision body pose detection occurs.
/// Transfers the sample data from the AVAssetReaderOutput to the AVAssetWriterInput,
/// processing via a CMSampleBufferProcessor.
///
/// - Parameters:
/// - readerOutput: The source sample data.
/// - writerInput: The destination for the sample data.
/// - queue: The DispatchQueue.
/// - completionHandler: The completion handler to run when the transfer finishes.
/// - Tag: transferSamplesAsynchronously
private func transferSamplesAsynchronously(from readerOutput: AVAssetReaderOutput,
to writerInput: AVAssetWriterInput,
onQueue queue: DispatchQueue,
sampleBufferProcessor: SampleBufferProcessor,
completionHandler: @escaping () -> Void) {
/*
The writerInput continously invokes this closure until finished or
cancelled. It throws an NSInternalInconsistencyException if called more
than once for the same writer.
*/
writerInput.requestMediaDataWhenReady(on: queue) {
var isDone = false
/*
While the writerInput accepts more data, process the sampleBuffer
and then transfer the processed sample to the writerInput.
*/
while writerInput.isReadyForMoreMediaData {
if self.isCancelled {
isDone = true
break
}
// Get the next sample from the asset reader output.
guard let sampleBuffer = readerOutput.copyNextSampleBuffer() else {
// The asset reader output has no more samples to vend.
isDone = true
break
}
// Process the sample, if requested.
do {
try sampleBufferProcessor?(sampleBuffer)
} catch {
/*
The `readingAndWritingDidFinish()` function picks up this
error.
*/
self.sampleTransferError = error
isDone = true
}
// Append the sample to the asset writer input.
guard writerInput.append(sampleBuffer) else {
/*
The writer could not append the sample buffer.
The `readingAndWritingDidFinish()` function handles any
error information from the asset writer.
*/
isDone = true
break
}
}
if isDone {
/*
Calling `markAsFinished()` on the asset writer input does the
following:
1. Unblocks any other inputs needing more samples.
2. Cancels further invocations of this "request media data"
callback block.
*/
writerInput.markAsFinished()
/*
Tell the caller the reader output and writer input finished
transferring samples.
*/
completionHandler()
}
}
}
The processor closure runs body pose detection on every sample buffer so that later in the VNDetectHumanBodyPoseRequest completion handler, VNHumanBodyPoseObservation results are fed into a custom Core ML action classifier.
private func videoProcessorForActivityClassification() -> SampleBufferProcessor {
let videoProcessor: SampleBufferProcessor = { sampleBuffer in
do {
let requestHandler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer)
try requestHandler.perform([self.detectHumanBodyPoseRequest])
} catch {
print("Unable to perform the request: \(error.localizedDescription).")
}
}
return videoProcessor
}
How could I improve the performance of this pipeline? After testing with an hour long 4K video at 60 FPS, it took several hours to process running as a Mac Catalyst app on M1 Max.
Post not yet marked as solved
I have a mlmodel prediction function in the script. During building (ios 15), the error shows up, and the app crashes. The whole build works under ios 14. The current Xcode version is 13.
Post not yet marked as solved
I'm trying to implement a pytorch custom layer [grid_sampler] (https://pytorch.org/docs/1.9.1/generated/torch.nn.functional.grid_sample.html) on GPU. Both of its inputs, input and grid can be 5-D. My implementation of encodeToCommandBuffer, which is MLCustomLayer protocol's function, is shown below. According to my current attempts, both value of id<MTLTexture> input and id<MTLTexture> grid don't meet expectations. So i wonder can MTLTexture be used to store 5-D input tensor as inputs of encodeToCommandBuffer? Or can anybody help to show me how to use MTLTexture correctly here? Thanks a lot!
- (BOOL)encodeToCommandBuffer:(id<MTLCommandBuffer>)commandBuffer
inputs:(NSArray<id<MTLTexture>> *)inputs
outputs:(NSArray<id<MTLTexture>> *)outputs
error:(NSError * _Nullable *)error {
NSLog(@"Dispatching to GPU");
NSLog(@"inputs count %lu", (unsigned long)inputs.count);
NSLog(@"outputs count %lu", (unsigned long)outputs.count);
id<MTLComputeCommandEncoder> encoder = [commandBuffer
computeCommandEncoderWithDispatchType:MTLDispatchTypeSerial];
assert(encoder != nil);
id<MTLTexture> input = inputs[0];
id<MTLTexture> grid = inputs[1];
id<MTLTexture> output = outputs[0];
NSLog(@"inputs shape %lu, %lu, %lu, %lu", (unsigned long)input.width, (unsigned long)input.height, (unsigned long)input.depth, (unsigned long)input.arrayLength);
NSLog(@"grid shape %lu, %lu, %lu, %lu", (unsigned long)grid.width, (unsigned long)grid.height, (unsigned long)grid.depth, (unsigned long)grid.arrayLength);
if (encoder)
{
[encoder setTexture:input atIndex:0];
[encoder setTexture:grid atIndex:1];
[encoder setTexture:output atIndex:2];
NSUInteger wd = grid_sample_Pipeline.threadExecutionWidth;
NSUInteger ht = grid_sample_Pipeline.maxTotalThreadsPerThreadgroup / wd;
MTLSize threadsPerThreadgroup = MTLSizeMake(wd, ht, 1);
MTLSize threadgroupsPerGrid = MTLSizeMake((input.width + wd - 1) / wd, (input.height + ht - 1) / ht, input.arrayLength);
[encoder setComputePipelineState:grid_sample_Pipeline];
[encoder dispatchThreadgroups:threadgroupsPerGrid threadsPerThreadgroup:threadsPerThreadgroup];
[encoder endEncoding];
}
else
return NO;
*error = nil;
return YES;
}
AncientCoder.
