Frames dropped after too many VNRecognizeTextRequest and VNDetectBarcodesRequest are executed under few seconds

Question

Created Nov ’22

Replies 1

Boosts 0

Views 1.1k

Participants 2

I am developing an iOS application to recognize texts and barcodes with the camera using VNRecognizeTextRequest and VNDetectBarcodesRequest.

The application performs loop detections and I am facing a frame dropped issue when 13 VNRecognizeTextRequest and VNDetectBarcodesRequest are performed in a row under few seconds. Before the frames are dropped, I do not see any errors during the 13 detections performed. When frames are dropped it is for an undetermined period of time but the detection manages to start again after.

Does anyone experienced this problem before ? Is there a hardware limitation that does not allow to perform more than 13 detections in a row to preserve the device battery or limit the use of resources ?

I already noticed that frame were dropped when the analysis frequency was lower than the image arrival frequency. This is understandable since the buffer queue is a FIFO. For example when I test the application on an old iPad since the analysis frequency is too low (in average 1 detection is performed in 1.2 seconds), frames are dropped during the analysis but the problem with the 13 detections does not occur.

However, I do not understand why a too high analysis frequency is a problem.

Thanks in advance

Answer 1

Technology Evangelist OP

Apple

Nov ’22

The most common cause of camera frame drops when doing recognition through Vision is how you're handling the camera buffers. When you get a frame from the camera, you want to initiate the Vision request and otherwise not retain it in any way. The camera system aggressively flushes frame buffers to ensure optimal performance. Also, it's highly recommended that you avoid detection on every camera frame. It's rarely necessary. Performing the Vision requests on something like every 5th or 10th frame should get you the app experience you want with less burden on the system—therein improving overall responsiveness.