gpuEvent related to IOSurface

Hi,

We have an AR project (built in Unity) that's running 24/7 on a number of iPads Pros (12.9, gen 3) in a museum. It's been live for 2.5 years now. We've always had the occasional freeze (on iOS 13 and below), but lately they have gotten exponentially worse (on iOS 14.4.2 and 14.6).

The problem manifests as follows, at completely random times: our app keeps running in the background (we can hear sound effects and see database activity continuing), but it stops getting redrawn, causing the screen to freeze on whatever frame was drawn last.

We're getting a gpuevent .ips log at freeze time (see below), which has information about an IOSurface being blocked by IOFence. There's very little info online about these logs though, so we're sort of stuck with the analysis. Can anyone here interpret this log better than we can, or does anyone have suggestions as to our next steps?

{"bug_type":"284","timestamp":"2021-07-10 10:51:30.00 +0200","os_version":"iPhone OS 14.6 (18F72)","incident_id":"C3ABD477-2A61-4D43-A3B1-25D7CCD706E6"}
{
  "process_name" : "hats",
  "registers" : {

  },
  "timestamp" : 1625907090,
  "analysis" : {
    "iofence_list" : {
      "iofence_num_iosurfaces" : 1,
      "iofence_iosurfaces" : [
        {
          "iofence_current_queue" : [
            {
              "iofence_acceleratorid" : 1,
              "iofence_backtrace" : [
                -68148358856,
                -68148357636,
                -68144533400,
                -68144532380,
                -68144496896,
                -68144598792,
                -68144425788,
                -68144434312
              ],
              "iofence_direction" : 1
            }
          ],
          "iosurface_id" : 243,
          "iofence_waiting_queue" : [
            {
              "iofence_acceleratorid" : 2,
              "iofence_backtrace" : [
                -68148358856,
                -68148357636,
                -68146420632,
                -68154559340,
                -68146423972,
                -68146427004,
                -68146495056,
                -68154290536
              ],
              "iofence_direction" : 2
            },
            {
              "iofence_acceleratorid" : 1,
              "iofence_backtrace" : [
                -68148358856,
                -68148357636,
                -68144533400,
                -68144532380,
                -68144496896,
                -68144598792,
                -68144425788,
                -68144434312
              ],
              "iofence_direction" : 1
            }
          ]
        }
      ]
    },
    "fw_ta_substate" : {
      "slot0" : 0,
      "slot1" : 0
    },
    "fw_power_state" : 0,
    "fw_power_boost_controller" : 0,
    "guilty_dm" : 1,
    "fw_power_controller_in_charge" : 0,
    "fw_cl_state" : {
      "slot0" : 0
    },
    "fw_perf_state_lo" : 1,
    "fw_ta_state" : {
      "slot0" : 0,
      "slot1" : 0
    },
    "signature" : 625,
    "fw_power_substate" : 4,
    "command_buffer_trace_id" : 490782964,
    "fw_perf_state_select" : 0,
    "restart_reason" : 7,
    "fw_3d_state" : {
      "slot0" : 0,
      "slot1" : 0,
      "slot2" : 0
    },
    "fw_gpc_perf_state" : 0,
    "fw_perf_state_hi" : 1,
    "fw_power_limit_controller" : 7,
    "restart_reason_desc" : "blocked by IOFence"
  }
}

I've posted a follow-up question here: https://developer.apple.com/forums/thread/685187

gpuEvent related to IOSurface
 
 
Q