A20 pro devices take too long on CoreML model specialization

Question

therysin OP

Created 3w

Replies 5

Boosts 6

Views 1.6k

Participants 4

Hi, I'm a ML developer specializing in local audio related models.

I upgraded from a 16 pro max to an 18 pro max.

I'm not sure what the reason is, but model specialization (happens only on the first load, cpu and neural engine compute units) is sometimes 2x - 5x longer on the 18 pro max vs the 16 pro max. Both devices are on iOS 27.

For example, I have a stem separation model running, and the 18 pro max took 171 seconds (including inference) and the 16 pro max took 37 seconds (including inference). 18 pro max inference is ~55x RTFx and the 16 pro max is ~40 RTFx for this particular model, but I've noticed it as well in TTS models etc. Please look into this, I know the 18 pro max has double the neural engine cores, so I'm not sure if the complexity makes specialization take longer.

Subsequent loads are great, same high rtfx for the 18 PM (only 0.5s load), and this is where it beats the 16 PM handily. I attached 2 images showing the 18 PM on a first load (model specialization likely) vs a subsequent run.

Please look into this. (Note this is not just my implementation, I tested a range of different models, all with the same outcome).

Answer 1

Nishant_Jaiswal OP

1w

Try using 32-bit version of the model. I had the same issue, and was using 16-bit model but when switched to 32-bit the model loading time was faster on 18 Pro

Answer 2

Htunkoko OP

1w

"I have experienced a similar issue with model specialization taking longer than expected. Clearing the cache or optimizing the CoreML model structure might help improve the processing time."

Answer 3

Htunkoko OP

1w

"I have experienced a similar issue with model specialization taking longer than expected. Clearing the cache or optimizing the CoreML model structure might help improve the processing time."

Answer 4

rsebbe OP

6d

same issue, MLModel loading with ANE hangs for 60-120s on iPhone 18 Pro, iOS 27.0.1. The result is cached though, and next loads are immediate.

This is a major problem though, that causes a poor first experience with the app. All other iPhones load it in <0.5s.

Answer 5

rsebbe OP

4d

Some additional testing, sharing the results:

I upgraded the iPhone 18 Pro to iOS 27.2 (Beta). The ANE compilation goes from about 120s to 20s under that OS. This is an improvement, but still a lot longer than all other iPhone, typically <1s.

I tested more of our models to see the impact that this could have. Out of 4 models tested, 2 presented that abnormally long ANE compilation time on the iPhone 18 Pro. I suspect some layer types cause this (no time unfortunately to investigate that, and retrain models). Note that it already occurs on very small models (<2MB), which, IMO, reveals this is a problem on the compiler that goes through unbounded computations when there's not much to optimize.

This is a real problem, as in our testing, ANE is useful, we can get as much as 4x performance increase (.all vs .cpuAndGPU loading), and the latest iPhone hardware shouldn't use a fallback that makes it slower than previous device generations.

Related: I noticed that MLModel device-based analysis in Xcode 27 (27.0, 27.1, 27.2) does not work anymore (load time, inference time, etc) at least for .mlmodel and .mlpackage (Core AI not tested). Is this a known situation?