Large memory consumption when running Core ML model on A13 GPU

Question

Created Dec ’22

Replies 3

Boosts 1

Views 1.9k

Participants 3

We recently had to change our MLModel's architecture to include custom layers, which means the model can't run on the Neural Engine anymore.

After the change, we observed a lot of crashes being reported on A13 devices. It turns out that the memory consumption when running the prediction with the new model on the GPU is much higher than before, when it was running on the Neural Engine. Before, the peak memory load was ~350 MB, now it spikes over 2 GB, leading to a crash most of the time. This only seems to happen on the A13. When forcing the model to only run on the CPU, the memory consumption is still high, but the same as running the old model on the CPU (~750 MB peak). All tested on iOS 16.1.2.

We profiled the process in Instruments and found that there are a lot of memory buffers allocated by Core ML that are not freed after the prediction.

The allocation stack trace for those buffers is the following:

We ran the same model on a different device and found the same buffers in Instruments, but there they are only 4 KB in size. It seems, Core ML is somehow massively over-allocating memory when run on the A13 GPU.

So far we limit the model to only run on CPU for those devices, but this is far from ideal. Is there any other model setting or workaround that we can use to avoid this issue?

Answer 1

FrankSchlegel OP

Dec ’22

Also reported as FB11871301. Thanks!

Answer 2

gudgud96 OP

Sep ’24

Hi @FrankSchlegel , I am facing the same issue whereby memory allocation with .cpuAndGpu is much higher than .cpuOnly. Also have the same observation that memory is not properly deallocated when running on GPU, even with some autoreleasepool tricks. Do you have any updates on this?

Answer 3

e_elephant OP

23h

I am running a CoreML model (vision transformer, fixed input size) with hardware units cpuAndNeuralEngine on both iPhone 15 and iPhone SE 2. The memory usage is normal on iPhone 15. On iPhone SE 2 [A13], it is more than double. The model loading on SE2 takes forever, and puts a few of these warnings in the console:

[Espresso::handle_ex_plan] exception=ANECF error: failed to load ANE model
[...]
FAILED: err=(
    CompilationFailure
)
doUnloadModel:options:qos:error:: nil _ANEModel
ANECF error:
Unable to build plan. @ PrepareOpForEncode
E5RT: Unable to build plan. (2)

Before seemingly falling back on CPU because inference completes successfully. But it looks as though the failed attempt at Espresso compilation leaves behind a ~800MB memory leak.

What's interesting is, if I set CoreML to use cpuOnly and not cpuAndNeuralEngine, the memory usage is lower by 800MB to around ~100MB, less than what I would expect.