function metadataOutput of protocol AVPlayerItemMetadataOutputPushDelegate is not always being triggered

Dear Sir/Madam,

Now I am working on a project that carry private information through ID3.PRIV field within HLS stream and expected get the information from function metadataOutput of protocol AVPlayerItemMetadataOutputPushDelegate. But there is a tricky situation when I played. metadataOutput is not always triggered. Sometimes it happens, sometime not. However, web player, Video.js could always get the data when it played. I am wondering what's the difference between AVPlayer and Video.js.

For this issue, I took some experiments.

First, I took basic HLS stream(https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8) which carry a ID3 data every 5 seconds. I found that every time the player launched, metadataOutput will be triggered as expected and I found that in that segments, audio, video and data stream will have the same pts at the beginning of the segment. I am curious is that the root cause of the problem. That's the requirement of AVPlayer.

Because in my own hls stream, audio, data and video will have different pts. If I played audio playlist only, the data will show the start time which is the offset between audio track and data track. Function metadataOutput will be triggered every time. But if I played master playlist which includes audio playlist and video playlists, it doesn't be triggered every time. When it is luckily triggered, the metadata start time show the offset between data track and video track.

Could anyone point out what's the problem and the normal behavior of pts sync when I use A/V split HLS stream.

All the information are taken as pictures and the links are included in the attachment.

Noted: In one situation, data pts is the same as video pts, and audio pts start from 0. Every time I use power button to turn off screen and turn on again, the function metadataOutput will be called immediately, but I don't know why.



Accompany with my sample code

Code Block swift
import UIKit
import AVFoundation
class ViewController: UIViewController, AVPlayerItemMetadataOutputPushDelegate {
  private var player: AVPlayer!
  private var playerLayer: AVPlayerLayer!
  private var playerItem: AVPlayerItem!
   
  @IBOutlet weak var videoView: UIView!
   func prepareToPlay(url: URL) {
    let asset = AVAsset(url: url)
    playerItem = AVPlayerItem(asset: asset)
    player = AVPlayer(playerItem: playerItem)
     
    playerLayer.player = player
    let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
    metadataOutput.setDelegate(self, queue: DispatchQueue.main)
    playerItem.add(metadataOutput)
    player.play()
  }
   
  override func viewDidLoad() {
    super.viewDidLoad()
    playerLayer = AVPlayerLayer(player: player)
    playerLayer.frame = self.videoView.bounds
    playerLayer.videoGravity = .resizeAspect
    self.videoView.layer.addSublayer(playerLayer)
     
    if let url = URL(string: "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8") {
     prepareToPlay(url: url)
    }
  }
  @available(iOS 8.0, *)
  func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) {
    print("metadata, groups : \(groups[0].timeRange)")
    if let item = groups.first?.items.first
    {
      print("item:\(item)")
      item.value(forKeyPath: #keyPath(AVMetadataItem.value))
      print("raw Metadata value: \n \(item.value(forKeyPath: #keyPath(AVMetadataItem.value))!)")
  }
  }



Change to a better way to make sure asset and playeritem is ready to play, then play.

But in these codes, I missed ID3 of first segment which is time 0 but got the following ID3 data from 5s and so on.

It only happen once for about 50 times of tests, possibility is really low. But I still wonder why there is this situation.

But from my own HLS stream , problem still exist.

import UIKit
import AVFoundation

class ViewController: UIViewController, AVPlayerItemMetadataOutputPushDelegate {

  private let player = AVPlayer()
  private var playerItemContext = 0
   
  @IBOutlet weak var videoView: UIView!
  @IBOutlet weak var playAndPauseButton: UIButton!
   
  @IBAction func playAndPause(_ sender: Any) {
    if (player.timeControlStatus == .paused) {
      playAndPauseButton.setImage(UIImage(systemName: "pause.rectangle.fill"), for: .normal)
      player.play()
    } else if (player.timeControlStatus == .playing) {
      playAndPauseButton.setImage(UIImage(systemName: "play.rectangle.fill"), for: .normal)
      player.pause()
    } else {
      print("unexpected timeControl status :\(player.timeControlStatus.rawValue)")
    }
  }
   
  override func observeValue(forKeyPath keyPath: String?,
                of object: Any?,
                change: [NSKeyValueChangeKey : Any]?,
                context: UnsafeMutableRawPointer?) {
    // Only handle observations for the playerItemContext
    guard context == &playerItemContext else {
      super.observeValue(forKeyPath: keyPath,
                of: object,
                change: change,
                context: context)
      return
    }

    if keyPath == #keyPath(AVPlayerItem.status) {
      let status: AVPlayerItem.Status

      // Get the status change from the change dictionary
      if let statusNumber = change?[.newKey] as? NSNumber {
        status = AVPlayerItem.Status(rawValue: statusNumber.intValue)!
      } else {
        status = .unknown
      }

      // Switch over the status
      switch status {
      case .readyToPlay:
        print("ready to play")
        self.playAndPause(self)
        break
      // Player item is ready to play.
      case .failed: break
      // Player item failed. See error.
      case .unknown: break
        // Player item is not yet ready.
      }
    }
  }
   
   
   
  private func initPlayerAsset(with url: URL, completion: ((_ asset: AVAsset) -> Void)?) {
    let asset = AVAsset(url: url)
    asset.loadValuesAsynchronously(forKeys: ["playable"]) {
      completion?(asset)
    }
  }
  func prepareToPlay(url: URL) {
    initPlayerAsset(with: url, completion: { (asset: AVAsset) in
      let playerItem = AVPlayerItem(asset: asset)

      let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
      metadataOutput.setDelegate(self, queue: DispatchQueue.main)
      playerItem.add(metadataOutput)

      playerItem.addObserver(self,
                  forKeyPath: #keyPath(AVPlayerItem.status),
                  options: [.old, .new],
                  context: &self.playerItemContext)
       
      self.player.replaceCurrentItem(with: playerItem)
    })
  }
   
  override func viewDidLoad() {
    super.viewDidLoad()

    let playerLayer = AVPlayerLayer(player: player)
    playerLayer.frame = self.videoView.bounds
    playerLayer.videoGravity = .resizeAspect

    self.videoView.layer.addSublayer(playerLayer)
     
    if let url = URL(string: "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8") {
      prepareToPlay(url: url)
    }
  }

  @available(iOS 8.0, *)
  func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) {
    print("metadata, groups : \(groups[0].timeRange)")
    if let item = groups.first?.items.first
    {
      print("item:\(item)")
      item.value(forKeyPath: #keyPath(AVMetadataItem.value))
      print("raw Metadata value: \n \(item.value(forKeyPath: #keyPath(AVMetadataItem.value))!)")
    } else {
      print("MetaData Error")
    }
  }
}

After weeks of try and verification, found that it I insert metadata track in video segment will not raise this problem. Only when m3u8 file is A/V split and metadata is in audio track, this callback will not be triggered every time. I am wondering if there is the reason that there are audio and video segments when assign AVPlayerItemOutput to player item and the binding may not work as expected for both audio and video

function metadataOutput of protocol AVPlayerItemMetadataOutputPushDelegate is not always being triggered
 
 
Q