VideoToolbox Decoding Performance Regression

I have created three related Feedback Assistant issues that haven't been replied to, and I also have found many WebKit bugs entered over the past six months that could be related to this issue (see links below).

  • FB9688897
  • FB9666426
  • FB9554184

Replication in iPadOS:

  1. Download - http://files.panomoments.com/bbb_sunflower_2160p_60fps_normal.mp4
  2. Attempt to play locally stored on the device either using Files app or Safari.
  3. Note the first few seconds playback with many frame drops and pauses (sound is unaffected).
  4. After initial playback, try seeking to various places in the timeline and note frame drops and stuttering.

Note - I am testing on a 1st Gen iPad Pro 13in

Replication in MacOS:

  1. Download - http://files.panomoments.com/bbb_sunflower_2160p_60fps_normal.mp4
  2. Note when opening in Quicktime the first frame is black.
  3. This also happens in the Finder spacebar preview function, but is harder to see.

The reason why you don’t usually see it in the spacebar preview, is likely because the preview video player has already decoded several frames asynchronously, and you just miss them due to the loading time of the UI. It’s a very fast flicker that’s easy to just ignore (unlike the frozen black frame in Quicktime Player)

Note - I am testing on a 2018 Macbook Pro i9

Regarding potentially related WebKit issues (it seems there was a ton of video decoding / GPU / WebGL work in iOS 15 and Safari 15), see these links:

https://bugs.webkit.org/show_bug.cgi?id=223740 https://bugs.webkit.org/show_bug.cgi?id=231031 https://bugs.webkit.org/show_bug.cgi?id=216250 https://bugs.webkit.org/show_bug.cgi?id=215908 https://bugs.webkit.org/show_bug.cgi?id=230617 https://bugs.webkit.org/show_bug.cgi?id=231359 https://bugs.webkit.org/show_bug.cgi?id=231424 https://bugs.webkit.org/show_bug.cgi?id=231012 https://bugs.webkit.org/show_bug.cgi?id=227586 https://bugs.webkit.org/show_bug.cgi?id=231354