HLS text track captions displaying at the wrong time

I have a video player application and while playing an HLS video on AVPlayer, I noticed that the muxed VTT subtitles are not showing up at the correct time.

For example, using AVPlayerItemLegibleOutput I can see that a text that was supposed to show at 24:37.107 is actually displaying at 24:36.406.

What I would like to know is if there's anything that I can do to identify why that is happening and what would cause AVPlayer to lose sync for the embedded captions.