Hi! I've just got a MacBook Air with M2 chip. I'm doing some research on ML and I was wondering if it is possible to use Neural Engine as an accelerator for training. If it is possible, where can I find ressources on how to enable it and how to check it? Thank you!

Build and installed Jax and Jax-metal following instructions on a M2Pro Mac-mini from here - https://developer.apple.com/metal/jax/ However, the following check seems to suggest XLA using CPU and not GPU. >>> from jax.lib import xla_bridge >>> print(xla_bridge.get_backend().platform) cpu Has anyone got it working to dump GPU? Thanks in advance!

I'm trying to use the randomTensor function from MPS graph to initialize the weights of a fully connected layer. I can create the graph and run inference using the randomly initialized values, but when I try to train and update these randomly initialized weights, I'm hitting a crash: Assertion failed: (isa<To>(Val) && "cast<Ty>() argument of incompatible type!"), function cast, file Casting.h, line 578. I can train the graph if I instead initialize the weights myself on the CPU, but I thought using the randomTensor functions would be faster/allow initialization to occur on the GPU. Here's my code for building the graph including both methods of weight initialization: func buildGraph(variables: inout [MPSGraphTensor]) -> (MPSGraphTensor, MPSGraphTensor, MPSGraphTensor, MPSGraphTensor) { let inputPlaceholder = graph.placeholder(shape: [2], dataType: .float32, name: nil) let labelPlaceholder = graph.placeholder(shape: [1], name: nil) // This works for inference but not training let descriptor = MPSGraphRandomOpDescriptor(distribution: .uniform, dataType: .float32)! let weightTensor = graph.randomTensor(withShape: [2, 1], descriptor: descriptor, seed: 2, name: nil) // This works for inference and training // let weights = [Float](repeating: 1, count: 2) // let weightTensor = graph.variable(with: Data(bytes: weights, count: 2 * MemoryLayout<Float32>.size), shape: [2, 1], dataType: .float32, name: nil) variables += [weightTensor] let output = graph.matrixMultiplication(primary: inputPlaceholder, secondary: weightTensor, name: nil) let loss = graph.softMaxCrossEntropy(output, labels: labelPlaceholder, axis: -1, reuctionType: .sum, name: nil) return (inputPlaceholder, labelPlaceholder, output, loss) } And to run the graph I have the following in my sample view controller: override func viewDidLoad() { super.viewDidLoad() var variables: [MPSGraphTensor] = [] let (inputPlaceholder, labelPlaceholder, output, loss) = buildGraph(variables: &variables) let gradients = graph.gradients(of: loss, with: variables, name: nil) let learningRate = graph.constant(0.001, dataType: .float32) var updateOps: [MPSGraphOperation] = [] for (key, value) in gradients { let updates = graph.stochasticGradientDescent(learningRate: learningRate, values: key, gradient: value, name: nil) let assign = graph.assign(key, tensor: updates, name: nil) updateOps += [assign] } let commandBuffer = MPSCommandBuffer(commandBuffer: Self.commandQueue.makeCommandBuffer()!) let executionDesc = MPSGraphExecutionDescriptor() executionDesc.completionHandler = { (resultsDictionary, nil) in for (key, value) in resultsDictionary { var output: [Float] = [0] value.mpsndarray().readBytes(&output, strideBytes: nil) print(output) } } let inputDesc = MPSNDArrayDescriptor(dataType: .float32, shape: [2]) let input = MPSNDArray(device: Self.device, descriptor: inputDesc) var inputArray: [Float] = [1, 2] input.writeBytes(&inputArray, strideBytes: nil) let source = MPSGraphTensorData(input) let labelMPSArray = MPSNDArray(device: Self.device, descriptor: MPSNDArrayDescriptor(dataType: .float32, shape: [1])) var labelArray: [Float] = [1] labelMPSArray.writeBytes(&labelArray, strideBytes: nil) let label = MPSGraphTensorData(labelMPSArray) // This runs inference and works // graph.encode(to: commandBuffer, feeds: [inputPlaceholder: source], targetTensors: [output], targetOperations: [], executionDescriptor: executionDesc) // // commandBuffer.commit() // commandBuffer.waitUntilCompleted() // This trains but does not work graph.encode( to: commandBuffer, feeds: [inputPlaceholder: source, labelPlaceholder: label], targetTensors: [], targetOperations: updateOps, executionDescriptor: executionDesc) commandBuffer.commit() commandBuffer.waitUntilCompleted() } And a few other relevant variables are created at the class scope: let graph = MPSGraph() static let device = MTLCreateSystemDefaultDevice()! static let commandQueue = device.makeCommandQueue()! How can I use these randomTensor functions on MPSGraph to randomly initialize weights for training?

Hello, I have my largely iOS app running using Mac Catalyst, but I need to limit what Macs will be able to install it from the Mac App Store based on the GPU Family like MTLGPUFamily.mac2. Is that possible? Or I could limit it to Apple Silicon using the Designed for iPad target, but I would prefer to use Mac Catalyst instead of Designed for iPad. Is it possible to limit Mac Catalyst installs to Apple Silicon Macs? Side question: what capabilities are supported by MTLGPUFamily.mac2? I can't find it. My main interest is in CoreML inference acceleration. Thank you.

I am performing a grid search over a parameter grid and train the model with different combinations of hyperparameters. I am receiving the following Warning: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz Why is that and what can I do to fix it? Thank you very much. Here is the code: def grid_search(model_name): ... elif model_name == 'LSTM': def build_model(units, activation, dropout, layers): model = Sequential() model.add(LSTM(units=units, kernel_initializer="normal", activation=activation, return_sequences=True, input_shape=(2, 1152), recurrent_dropout=0)) model.add(Dropout(dropout)) for i in range(layers): if i != layers-1: model.add(LSTM(units=units, kernel_initializer="normal", activation=activation, return_sequences=True,recurrent_dropout=0)) model.add(Dropout(dropout)) elif i == (layers-1): model.add(LSTM(units=units, kernel_initializer="normal", activation=activation, recurrent_dropout=0)) model.add(Dropout(dropout)) model.add(Dense(units=6, kernel_initializer="normal", activation=activation)) model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]) return model param_grid = {'units': [200, 300, 400], 'activation': ['tanh'], 'dropout': [0, 0.2, 0.4, 0.6], 'layers': [0, 5]} group_kfold = GroupKFold(n_splits=len(np.unique(groups_train))) model = KerasClassifier(model=build_model, units=param_grid['units'], activation=param_grid['activation'], dropout=param_grid['dropout'], layers=param_grid['layers']) grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=group_kfold) X_test, X_train, y_test, y_train = raw_dataassigner(model_name) (X_train, y_train) = shuffle(X_train, y_train) with tf.device('/cpu:0'): grid_result = grid_search.fit(X_train, y_train, groups=groups_train) print(f'Best score ({grid_search.best_score_}) for {model_name} model achieved with parameters: ', grid_search.best_params_) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param)) grid_search('LSTM')

When I run the performance test on a CoreML model, it shows predictions are 834% faster running on the Neural Engine as it is on the GPU. It also shows, that 100% of the model can run on the Neural Engine: GPU only: But when I set the compute units to all: let config = MLModelConfiguration() config.computeUnits = .all and profile, it shows that the neural engine isn’t used at all. Well, other than loading the model which takes 25 seconds when allowed to use the neural engine versus less than a second when not allowing the neural engine: The difference in speed is the difference between the app being too slow to even release versus quite reasonable performance. I have a lot of work invested in this, so I am really hoping that I can get it to run on the Neural Engine. Why isn't it actually running on the Neural Engine when it shows that it is supported and I have the compute unit set to run on the Neural Engine?

Hi Guys, I am using a MacBook pro 2019 Intel core i7, 16inch 2019 model device. When i am using an external monitor my CPU spikes and Kernel_task is using more than 700% CPU. I am using the apple manufactured HDMI extender to connect to the monitor. As soon as i disconnect the monitor, everything works fine. I am running Ventura 13.3.1.

Hi, someone can tell me if it's possible and eventually how to see the Confusion Matrix about a CreateML Model? Thank you!

I know there are some operations that tensorflow-metal does not support yet. but i want to fit/execute model that I constructed for kg in gpu setting. So please make it possible to execute resource gather operation in gpu setting.

I have an app in the App Store that has been running fine. Starting yesterday it began throwing an exception during initialization in the App Store and during development: "[0x000...***] during in one-time initialization function for my_model_name at /ViewController.swift:84" Thread 2: EXC_BREAKPOINT (code=1, =***) [coreml] Could not create persistent key blob for ----*** : error=Error Domain=com.apple.CoreML Code=8 "Fetching decryption key from server failed." UserInfo=NSLocalizedDescription=Fetching decryption key from server failed., NSUnderlyingError=0x2822c6a60 Error Domain=CKErrorDomain Code=6 "CKInternalErrorDomain: 2022" UserInfo=NSDebugDescription=CKInternalErrorDomain: 2022, RequestUID=----, NSLocalizedDescription=Request failed with http status code 503, CKErrorDescription=Request failed with http status code 503, CKRetryAfter=21, NSUnderlyingError=0x2822c6ac0 Error Domain=CKInternalErrorDomain Code=2022 "Request failed with http status code 503" UserInfo=CKRetryAfter=21, CKHTTPStatus=503, CKErrorDescription=Request failed with http status code 503, RequestUID=----, NSLocalizedDescription=Request failed with http status code 503, CKHTTPStatus=503 ... NSLocalizedDescription=Request failed with http status code 503, CKHTTPStatus=503 ... Help! Anyone know what is going on? ... Also, when I try to generate a new key. I get this: I'm the only admin developer on my account. I tried following its directions by signing out and back in with my Apple ID in preferences. Restarted XCode and still can't generate a new key, but still received the same error.

When I use an mlmodel called tinyyolo in my project, after building, xcode reports “ Command CodeSign failed with a nonzero exit code”. The problem should be with the mlmodel. But I don't know what‘s wrong and how to solve it.

Hey! I have a question. I try to add some pictures in Create ML and it tells me that I'm missing something. What do I need to add? (I mention that in that folder there are pictures + a .json with all the pictures) If you have any recommendations.

Hi, I am trying to take advantage of my device ANE. I have created a model from torch using coremltools and adapted it until xcode model performance preview indicates it will run on my device ANE. But when i profile my integration into my app, i can see from the com.apple.ane logs the model has been loaded on device: Timestamp Type Process Category Message 00:00.905.087 Debug MLBench (11135) client doLoadModel:options:qos:error:: model[0x2804500c0] : success=1 : progamHandle=10 000 241 581 886: intermediateBufferHandle=10 000 242 143 532 : queueDepth=32 :err= but when i call predict on my model, the ANE is unloaded and the prediction run on CPU: Timestamp Type Process Category Message 00:00.996.015 Debug MLBench (11135) client doUnloadModel:options:qos:error:: model[0x2804500c0]=_ANEModel: { modelURL=file:///var/mobile/Containers/Data/Application/0A9F356B-B8C7-4B86-90A5-6812EF48CC94/tmp/math_custom_trans_decoder_seg_0DB63A47-E84E-4887-A606-BC9986B2C662.mlmodelc/ : key={"isegment":0,"inputs":{"extras":{"shape":[2,1,1,1,1]},"memory":{"shape":[128,5,1,1,1]},"proj_key_seg_in":{"shape":[128,39,1,1,1]},"state_in_k":{"shape":[32,1,1,20,2]},"tgt":{"shape":[5,1,1,1,1]},"state_in_v":{"shape":[32,1,1,20,2]},"pos_enc":{"shape":[128,1,1,1,1]}},"outputs":{"attn_seg":{"shape":[1,5,1,4,1]},"state_out_v":{"shape":[32,2,1,20,2]},"output":{"shape":[292,5,1,1,1]},"state_out_k":{"shape":[32,2,1,20,2]},"extras_tmp":{"shape":[2,1,1,1,1]},"proj_key_seg_in_tmp":{"shape":[128,39,1,1,1]},"attn":{"shape":[1,1,1,5,2]},"proj_key_seg":{"shape":[128,1,1,20,1]}}} : string_id=0x70ac000000015257 : program=_ANEProgramForEvaluation: { programHandle=10000241581886 : intermediateBufferHandle=10000242143532 : queueDepth=32 } : state=3 : programHandle=10000241581886 : intermediateBufferHandle=10000242143532 : queueDepth=32 : attr=... : perfStatsMask=0}  i dont see any obvious error messages in com.apple.ane, com.apple.coreml or com.apple.espresso. where/what should i look for to understand what is going on? and in particular why the ANE model was unloaded? Thank you

devices: iphone 11 config: configuration.computeUnits = .all let myModel = try! myCoremlModel(configuration: configuration).model Set the Range for Each Dimension: input_shape= ct.Shape(shape=(1,3,ct.RangeDim(lower_bound=128, upper_bound=384, default=256),ct.RangeDim(lower_bound=128, upper_bound=384, default=256))) inference time as table(average of 100 runs) The default size inference for dynamic models is the same as for static models, but 128128 and 384384 hundreds of times slow than fixed-size models. Is this normal? Is there any good solution? model init time is too long load model time about 2 minutes, Is there a way to speed it up? For example, load from the cache? Can converted mlparkage speed up the loading time?

I'm trying to run sample code for MPS graph, which I got here: https://developer.apple.com/documentation/metalperformanceshadersgraph/adding_custom_functions_to_a_shader_graph And it's not working. Builds successfully, but after you press train (play button), program fails right after first training iteration with errors like these: -[MTLDebugCommandBuffer lockPurgeableObjects]:2103: failed assertion `MTLResource 0x600001693940 (label: (null)), referenced in cmd buffer 0x124015800 (label: (null)) is in volatile or empty purgeable state at commit' -[MTLDebugCommandBuffer lockPurgeableObjects]:2103: failed assertion `MTLResource 0x600001693940 (label: (null)), referenced in cmd buffer 0x124015800 (label: (null)) is in volatile or empty purgeable state at commit' It is failing on commandBuffer.commit() in runTrainingIterationBatch() method. Its like something already committed operation (I've checked and yeah, command buffer is already commited). But why such thing in EXAMPLE CODE? I've tried to wrap commit operation with command buffer status check and it is helping to not fail, but program works wrong overall and not calculating loss well. Everything is getting worse because documentation for MPS Graph is empty! It's contains only class and method names without any description D; My env: Xcode 13.4.1 (13F100) macOS 12.4 MacBook Pro (m1 pro) 14' 2021 16gb Tried to build on iPhone 12 Pro Max / iOS 15.5 and to Mac catalyst application. Got same error everywhere

Is there any way to train PyTorch models directly on-device on an iPhone via the GPU? PyTorch Mobile docs seems to be completely focused on inference only, as do the iOS app examples (https://github.com/pytorch/ios-demo-app). I did find this article about using MPS backend on Macs (https://developer.apple.com/metal/pytorch/), but not sure if this is at all viable for iOS devices. There's also this prototype article about using iOS GPU for PyTorch mobile (https://pytorch.org/tutorials/prototype/ios_gpu_workflow.html), but it too seems to be focused on inference only. We are attempting to train a large language model on the iPhone 14 and we would like to a) discard intermediate activations and recompute them, and b) manage memory directly to write some intermediate activations to the filesystem and later read them back. We suspect that converting a PyTorch model to CoreML format and using CoreML for training would prevent us from making these low-level modifications, but PyTorch might have the APIs necessary for this. If there's any examples/pointers that anyone can link to that would be great.

I specifically give the model the shape of (1, 3, 1024, 1024), but for some reason, CoreML thinks it's 2 channels instead of 3. The pytorch model is based on this - LINK The "local.pth" model to be specific. My CoreML conversion code is attached below. #from networks.drn_seg import DRNSeg import coremltools as ct import coremltools.proto.FeatureTypes_pb2 as ft import io from PIL import Image from torchvision import transforms import math import torch import torch.nn as nn from networks.drn import drn_c_26 import torchvision from torchvision.io import read_image def fill_up_weights(up): w = up.weight.data f = math.ceil(w.size(2) / 2) c = (2 * f - 1 - f % 2) / (2. * f) for i in range(w.size(2)): for j in range(w.size(3)): w[0, 0, i, j] = \ (1 - math.fabs(i / f - c)) * (1 - math.fabs(j / f - c)) for c in range(1, w.size(0)): w[c, 0, :, :] = w[0, 0, :, :] class DRNSeg(nn.Module): def __init__(self): super(DRNSeg, self).__init__() classes=2 pretrained_drn=None pretrained_model=None use_torch_up=False model = drn_c_26(pretrained=pretrained_drn) self.base = nn.Sequential(*list(model.children())[:-2]) if pretrained_model: self.load_pretrained(pretrained_model) self.seg = nn.Conv2d(model.out_dim, classes, kernel_size=1, bias=True) m = self.seg n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels m.weight.data.normal_(0, math.sqrt(2. / n)) m.bias.data.zero_() if use_torch_up: self.up = nn.UpsamplingBilinear2d(scale_factor=8) else: up = nn.ConvTranspose2d(classes, classes, 16, stride=8, padding=4, output_padding=0, groups=classes, bias=False) fill_up_weights(up) up.weight.requires_grad = False self.up = up def forward(self, x): x = self.base(x) x = self.seg(x) y = self.up(x) return y def optim_parameters(self, memo=None): for param in self.base.parameters(): yield param for param in self.seg.parameters(): yield param def load_pretrained(self, pretrained_model): print("loading the pretrained drn model from %s" % pretrained_model) state_dict = torch.load(pretrained_model, map_location='cpu') if hasattr(state_dict, '_metadata'): del state_dict._metadata # filter out unnecessary keys pretrained_dict = state_dict['model'] pretrained_dict = {k[5:]: v for k, v in pretrained_dict.items() if k.split('.')[0] == 'base'} # load the pretrained state dict self.base.load_state_dict(pretrained_dict) #img = read_image('test.jpg') model_path = ("modelphotoshop.pth") device = 'cpu' model_flow = DRNSeg() model_flow.load_state_dict(torch.load(model_path, map_location=device)) model_flow.to(device) model_flow.eval() example_input = torch.randn([1,3,1024,1024]) traced_model = torch.jit.trace(model_flow, example_input) traced_model.save("modelphotoshop.pt") shape = (1,3, 1024, 1024) mlmodel = ct.convert(traced_model, convert_to="mlprogram", inputs=[ct.ImageType(name="colorImage", shape=shape, color_layout=ct.colorlayout.RGB,)], outputs=[ct.ImageType(name="colorOutput", color_layout=ct.colorlayout.RGB,)], ) mlmodel.save("Model.mlmodel")

Hello, Is there any way to make a program run on the Neural Engine? I have a compiled program (not Python/tensorflow/etc) that I would like to speed up; right now, it runs on the GPU but I was told by the developer it doesn’t use the neural engine. I’m running predictions, not training models - I have read that the ANE is especially good for this purpose. Is there a way to speed up my predictions here with the ANE? Noah

I have a 16-inch M1 pro with 32 GB ram and an Nvidia A6000 GPU. What is the best way to connect the GPU so I can use it with VS code? Thanks.

Hi from France ! I'm trying to create a model for dice detection. I've take about 100 photos of dice on the same side (1 point). Are-my bounding boxes good ? should I take the whole dice ? I launched the trainning, it seems to work well : Then in the Evaluation tab, the values seems not great but not bad : I/U 84% Varied I/U 44% The validation scope is very low : In the preview tab, no matter what image I give to it, I have no detection What am I missing ? What should I improve ?