Hi I have been the following WWDC21 "dynamic training on iOS" - I have been able to get the training working, with an output of the iterations etc being printed out in the console as training progresses.
However I am unable to retrieve the checkpoints or result/model once training has completed (or is in progress) nothing in the callback fires.
If I try to create a model from the sessionDirectory - it returns nil (even though training has clearly completed).
Please can someone help or provide pointers on how to access the results/checkpoints so that I can make a MlModel and use it.
var subscriptions = [AnyCancellable]()
let job = try! MLStyleTransfer.train(trainingData: datasource, parameters: trainingParameters, sessionParameters: sessionParameters)
job.result.sink { result in
print("result ", result)
}
receiveValue: { model in
try? model.write(to: sessionDirectory)
let compiledURL = try? MLModel.compileModel(at: sessionDirectory)
let mlModel = try? MLModel(contentsOf: compiledURL!)
}
.store(in: &subscriptions)
This also does not work:
job.checkpoints.sink { checkpoint in
// Process checkpoint
let model = MLStyleTransfer(trainingData: checkpoint)
}
.store(in: &subscriptions)
}
This is the printout in the console:
Using CPU to create model
+--------------+--------------+--------------+--------------+--------------+
| Iteration | Total Loss | Style Loss | Content Loss | Elapsed Time |
+--------------+--------------+--------------+--------------+--------------+
| 1 | 64.9218 | 54.9499 | 9.97187 | 3.92s |
2022-02-20 15:14:37.056251+0000 DynamicStyle[81737:9175431] [ServicesDaemonManager] interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke
| 2 | 61.7283 | 24.6832 | 8.30343 | 9.87s |
| 3 | 59.5098 | 27.7834 | 11.7603 | 16.19s |
| 4 | 56.2737 | 16.163 | 10.985 | 22.35s |
| 5 | 53.0747 | 12.2062 | 12.0783 | 28.08s |
+--------------+--------------+--------------+--------------+--------------+
Any help would be appreciated on how to retrieve models.
Thanks
Post not yet marked as solved
For a Create ML activity classifier, I’m classifying “playing” tennis (the points or rallies) and a second class “not playing” to be the negative class. I’m not sure what to specify for the action duration parameter given how variable a tennis point or rally can be, but I went with 10 seconds since it seems like the average duration for both the “playing” and “not playing” labels.
When choosing this parameter however, I’m wondering if it affects performance, both speed of video processing and accuracy. Would the Vision framework return more results with smaller action durations?
Post not yet marked as solved
i am using the tabular regression method of CreateML. i see where it prints a couple of metrics such as root mean square error on the validation data, but i dont see any way to export the raw fitted numbers (e.g. training prediction), or validation numbers (e.g. validation prediction), or out of sample "testing" numbers (from the testing data set). is this possible in CreateML directly? the reason this is necessary is that you sometimes want to plot actual versus predicted and compute other metrics for regressions.
Post not yet marked as solved
I need to build a model to add to my app and tried following the Apple docs here.
No luck because I get an error that is discussed on this thread on the forum. I'm still not clear on why the error is occurring and can't resolve it.
I wonder if CreateML inside Playgrounds is still supported at all? I tried using the CreateML app that you can access through developer tools but it just crashes my Mac (2017 MBP - is it just too much of a brick to use for ML at this point? I should think not because I've recently built and trained relatively simple models using Tensorflow. + Python on this machine, and the classifier I'm trying to make now is really simple and doesn't have a huge dataset).
Post not yet marked as solved
How should I think about video quality (if it's important) when gathering training videos? Does higher video quality of training data make for better predictions, or should it more closely match the common use case (1080p I suppose, thinking about iPhones broadly)?
Post not yet marked as solved
One part of my app uses CreateML but I still need to test the UI on other screen sizes with the simulator. I tried using this condition but still get the same error.
#if os(iOS)
import CreateML // CreateML is not available when building for iOS Simulator. Consider using `#if os(iOS)` to conditionally import this framework when building for iOS.
#endif
Post not yet marked as solved
I want to detect an image of a dart target (https://commons.wikimedia.org/wiki/File:WA_80_cm_archery_target.svg) in my iOS app.
For that I am creating an object detector with CreateML. I am using the Transfer Learning algorithm and 114 annotated images, the validation data is set to auto.
After 2000 iterations I got the following stats: 96% Training and 0% Validation.
As I understand it, the percentages are the I/U 50% scores (= percentage of intersection over union ratios from the bounding boxes with over 50%).
If the validation data is automatically chosen from the set of images, how can its score be 0%?
Post not yet marked as solved
I just got an app feature working where the user imports a video file, each frame is fed to a custom action classifier, and then only frames with a certain action classified are exported.
However, I'm finding that testing a one hour 4K video at 60 FPS is taking an unreasonably long time - it's been processing for 7 hours now on a MacBook Pro with M1 Max running the Mac Catalyst app. Are there any techniques or general guidance that would help with improving performance? As much as possible I'd like to preserve the input video quality, especially frame rate. One hour length for the video is expected, as it's of a tennis session (could be anywhere from 10 minutes to a couple hours). I made the body pose action classifier with Create ML.
Post not yet marked as solved
After creating a custom action classifier in Create ML, previewing it (see the bottom of the page) with an input video shows the label associated with a segment of the video. What would be a good way to store the duration for a given label, say, each CMTimeRange of segment of video frames that are classified as containing "Jumping Jacks?"
I previously found that storing time ranges of trajectory results was convenient, since each VNTrajectoryObservation vended by Apple had an associated CMTimeRange.
However, using my custom action classifier instead, each VNObservation result's CMTimeRange has a duration value that's always 0.
func completionHandler(request: VNRequest, error: Error?) {
guard let results = request.results as? [VNHumanBodyPoseObservation] else {
return
}
if let result = results.first {
storeObservation(result)
}
do {
for result in results where try self.getLastTennisActionType(from: [result]) == .playing {
var fileRelativeTimeRange = result.timeRange
fileRelativeTimeRange.start = fileRelativeTimeRange.start - self.assetWriterStartTime
self.timeRangesOfInterest[Int(fileRelativeTimeRange.start.seconds)] = fileRelativeTimeRange
}
} catch {
print("Unable to perform the request: \(error.localizedDescription).")
}
}
In this case I'm interested in frames with the label "Playing" and successfully classify them, but I'm not sure where to go from here to track the duration of video segments with consecutive frames that have that label.
Post not yet marked as solved
I followed Apple's guidance in their articles Creating an Action Classifier Model, Gathering Training Videos for an Action Classifier, and Building an Action Classifier Data Source. With this Core ML model file now imported in Xcode, how do use it to classify video frames?
For each video frame I call
do {
let requestHandler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer)
try requestHandler.perform([self.detectHumanBodyPoseRequest])
} catch {
print("Unable to perform the request: \(error.localizedDescription).")
}
But it's unclear to me how to use the results of the VNDetectHumanBodyPoseRequest which come back as the type [VNHumanBodyPoseObservation]?. How would I feed to the results into my custom classifier, which has an automatically generated model class TennisActionClassifier.swift? The classifier is for making predictions on the frame's body poses, labeling the actions as either playing a rally/point or not playing.
Post not yet marked as solved
My goal is to mark any tennis video's timestamps of both the start of each rally/point and the end of each rally/point. I tried trajectory detection, but the "end time" is when the ball bounces rather than when the rally/point ends. I'm not quite sure what direction to go from here to improve on this. Would action classification of body poses in each frame (two classes, "playing" and "not playing") be the best way to split the video into segments? A different technique?
Post not yet marked as solved
Beginner at using CreateML, so please forgive me if this question isn't asked correctly. As i understand it, the image classification projects are meant to detect certain objects in an image (giraffe vs elephant). My question is is there a way to use image classification to "score" or bin images that share qualities with my training dataset?
As an example; let's say I want to find a perfect square inside another square (like a white border around an image). What are the things that could make a "non-perfect" image? maybe one of the corners of the square is rounded, maybe a corner is not 90 degrees, maybe the inner square is not perfectly centered within the white frame / border.
Now let's say I want to take a picture of this object and have my app tell me how close this image is to a perfect square inside a square and rate them 1-5
My thought was to setup my training data to have a set of images that show perfect squares in a "rated 5" folder, a set of slightly imperfect squares in a "rated 4" folder, and a set of even less perfect squares in a "rated 3" folder, etc.
Long winded question, i apologize; will the CreateML image classifier be able to look at my image for those qualities that make them 3,4,5, or will it only look at the content of the square itself and detect: Giraffe, race car, boat, person? I'm looking agin for the metric of "perfectness" regardless of what the content is within the inner square. Am I on the right train of thought, or is there a better approach to take?
Post not yet marked as solved
After collecting data I wanted to create an ML model I have done this before but for some reason I was getting the error:
error: Couldn't lookup symbols:
CreateML.MLDataTable.init(contentsOf: Foundation.URL, options: CreateML.MLDataTable.ParsingOptions) throws -> CreateML.MLDataTable
CreateML.MLDataTable.init(contentsOf: Foundation.URL, options: CreateML.MLDataTable.ParsingOptions) throws -> CreateML.MLDataTable
So I went to test a working example that was created by apple using this link: https://developer.apple.com/documentation/createml/creating_a_model_from_tabular_data
After running this test with no data changed, I still get the same error logged. I don't know if I'm doing something wrong. Any advice would be greatly appreciated.
Post not yet marked as solved
I tried Create ML to train MNIST dataset which has very small images of 0-10 digits. It's the first time I use Create ML but its training speed is still too slow based on what I learnt, MNIST is a very small dataset.
I am using a MacBook Pro 2021, 16 inch, with M1 pro + 16GB ram + 1TB SSD.
I check the activity monitor and saw that CPU reaches 100%.
14/16 GB of Memory are used, 2GB for cache and 12.5GB of swap used. Memory used by the MLRecipeExecutionService process is 19.55GB. If I double click to see the details, the Virtual Memory Size is 410GB.
I ran sudo powermetrics and observe that GPU power is ~50-60mw, which means GPU is not used for training.
When I check Disk usage in Activity Monitor, I saw that process MLRecipeExecutionService contributed 1.1TB of Bytes Write. The entire MNIST dataset is only 17.5MB.
I don't understand why it's so slow, and so much resources were used. Based on what I've learnt about Machine Learning, this is irregular.
Post not yet marked as solved
Being brand new to create ML I tried to run my own ML project. When creating my own image classifier (same with tabular classification) I fail from the start. When selecting valid training data create ML says "Data Analysis stopped". I'm using Create ML Version 3.0 (78.7).
Any suggestions?
Post not yet marked as solved
I have trained a model with CreateML. If I test the results with the Preview option that comes with the mlmodel, it shows me some preditions with a given conficence, but if I go through Vision + CoreML to check the predictions, for the same images, the confidence is totally different.
Here is an example of the output, console output is from the playground with vision + CoreML and the image footer is from the preview of the model itself. I have sent this model to a colleague that uses Coremltools in Python and the results are also different.
Does the prediction affect where you are executing the model on?
Post not yet marked as solved
When trying to train an image classifier with Create ML I hit the train button and after the feature extracting phase, the training tab chart is empty
I have tried with different images and even training different models (one of them the typical dog vs cat model) but the result is the same, how can I get this to work?
Post not yet marked as solved
Hi,
is it possible to get the code for the demo app used in this presentation for the dynamic style transfer example please?
thanks
Post not yet marked as solved
Hi there, I'm trying to train an mlmodel using shorter file lengths. I'd like to not have a lower limit on the length of the audio. Is there any way to do this?
Post not yet marked as solved
I'm getting an error very early in the the process and these tutorials seems very simple so I'm stumped.
This tutorial seems straightforward but I can't make it past the step where I drag in image sets in.
https://developer.apple.com/documentation/createml/creating_an_image_classifier_model
video tutorial: https://www.youtube.com/watch?v=DSOknwpCnJ4
I have 1 folder titled "Training Data" with 2 sub-folders "img1" and "img2". When I drag my folder "Training Data" into the Training Data section I get the error: "No training data found. 0 invalid files found."
I have no idea what is causing this. Images are .jpg and taken from my phone. I only have 6 total images in the initial test. I've tried it with and without an annotations.json file created in COCO Annotator, that didn't make a difference same error with or without.
Big Sur 11.5.2
Create ML 3.0
