We have an AR project (built in Unity) that's running 24/7 on a number of iPads Pros (12.9, gen 3) in a museum. It's been live for 2.5 years now. We've always had the occasional freeze (on iOS 13 and below), but lately they have gotten exponentially worse (on iOS 14.4.2 and 14.6).
The problem manifests as follows, at completely random times: our app keeps running in the background (we can hear sound effects and see database activity continuing), but it stops getting redrawn, causing the screen to freeze on whatever frame was drawn last.
We get a gpuEvent log (see here), and when running the app when connected to XCode, we get the following error message a few times:
Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2)
followed by this one, which keeps repeating:
Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (IOAF code 4)
A couple of potentially related threads:
- https://developer.apple.com/forums/thread/73225
- https://forum.unity.com/threads/iphone-8-a11-gpu-hang-error-command-buffer.500549/
- https://github.com/KhronosGroup/MoltenVK/issues/602
Especially the last one seems to describe exactly the same problem as we're having.
We've tried to reproduce the freeze with XCode's API and Shader validation options, but that doesn't give a lot of readable info - possibly because we're building with Unity 2018.4? I don't know how to connect this info to the correct Unity shader file, for instance - let alone the offending line of code. Screenshot below.
Does anyone have any suggestions as to how we should continue?
