Is the inner under-display camera going to be usable for real-time 2D body-pose analysis?

Question

AlexanderJunger OP

Created 8h

Replies 1

Boosts 0

Participants 2

This post is from the Activity iPhone Duo Photos & Camera Q&A.

Hi!

We run continuous 2D human body-pose estimation on a front-facing camera feed: one person, full body in frame, standing a meter or two away, 30-60fps.

With iPhone Duo open, the camera that faces the person looking at the inner display is the under-display inner camera (builtInInnerUltraWideCamera). Published specs cap it at 1080p at 30 or 60 fps, and it is presented as the FaceTime camera.

Two things are unclear from the camera tech talk and the specs:

Is that 1080p format a native readout we can feed to Vision body pose, or a preview-oriented stream whose effective detail is too low for full-body pose at that distance?

Is third-party use of this camera supported for a long-running analysis session, or is it intended only for video calls and live previews?

Thanks

Answered by DTS Engineer in 906602022

Hi @AlexanderJunger,

You wrote:

Is that 1080p format a native readout we can feed to Vision body pose, or a preview-oriented stream whose effective detail is too low for full-body pose at that distance?

The inner under-display camera is represented in AVFoundation by the builtInInnerUltraWideCamera device type. Published specifications indicate that this camera supports a native readout format capped at 1080p at 30 or 60 fps. This format is a native camera readout suitable for ingestion into frameworks like Vision, rather than being limited to a low-detail preview stream.

Then, you wrote:

Is third-party use of this camera supported for a long-running analysis session, or is it intended only for video calls and live previews?

Third-party use of the builtInInnerUltraWideCamera for long-running analysis sessions, such as continuous human body-pose estimation, is fully supported when capturing from individual physical cameras. While the virtual front camera abstracts switching between physical lenses, capturing from specific individual devices grants access to their full capabilities. Coordinate camera direction changes using AVCaptureDeviceDirectionCoordinator when the device opens or closes. To learn more, see Choosing a camera by the direction it faces.

To perform continuous body pose estimation with Vision:

Set up an AVCaptureSession using builtInInnerUltraWideCamera as your video input.
Add an AVCaptureVideoDataOutput to receive video sample buffers from the live camera feed.
Process each sample buffer by creating a VNImageRequestHandler and performing a VNDetectHumanBodyPoseRequest to detect up to 19 unique body points.

For more details, see Detecting human body poses in images.

When processing a high-framerate feed (30-60fps) for continuous analysis, set alwaysDiscardsLateVideoFrames on your AVCaptureVideoDataOutput to true. This prevents frame backlogs, avoids starving the camera stream of buffers, and keeps the application operating smoothly in real time. To learn more, see Detecting human actions in a live video feed.

Cheers,

Paris X Pinkney |  WWDR | DTS Engineer

Answer 1

DTS Engineer OP

Apple

3h

Recommended

Hi @AlexanderJunger,

You wrote:

Is that 1080p format a native readout we can feed to Vision body pose, or a preview-oriented stream whose effective detail is too low for full-body pose at that distance?

The inner under-display camera is represented in AVFoundation by the builtInInnerUltraWideCamera device type. Published specifications indicate that this camera supports a native readout format capped at 1080p at 30 or 60 fps. This format is a native camera readout suitable for ingestion into frameworks like Vision, rather than being limited to a low-detail preview stream.

Then, you wrote:

Is third-party use of this camera supported for a long-running analysis session, or is it intended only for video calls and live previews?

Third-party use of the builtInInnerUltraWideCamera for long-running analysis sessions, such as continuous human body-pose estimation, is fully supported when capturing from individual physical cameras. While the virtual front camera abstracts switching between physical lenses, capturing from specific individual devices grants access to their full capabilities. Coordinate camera direction changes using AVCaptureDeviceDirectionCoordinator when the device opens or closes. To learn more, see Choosing a camera by the direction it faces.

To perform continuous body pose estimation with Vision:

Set up an AVCaptureSession using builtInInnerUltraWideCamera as your video input.
Add an AVCaptureVideoDataOutput to receive video sample buffers from the live camera feed.
Process each sample buffer by creating a VNImageRequestHandler and performing a VNDetectHumanBodyPoseRequest to detect up to 19 unique body points.

For more details, see Detecting human body poses in images.

When processing a high-framerate feed (30-60fps) for continuous analysis, set alwaysDiscardsLateVideoFrames on your AVCaptureVideoDataOutput to true. This prevents frame backlogs, avoids starving the camera stream of buffers, and keeps the application operating smoothly in real time. To learn more, see Detecting human actions in a live video feed.

Cheers,

Paris X Pinkney |  WWDR | DTS Engineer