How can an app detect a TrueDepth pipeline failure when no depth output or drop callback is delivered?

How can an app detect a TrueDepth pipeline failure when no depth output or drop callback is delivered?

We are using AVFoundation to stream synchronized RGB and depth data from the front-facing TrueDepth camera. On a small number of physical devices, the capture session and RGB stream remain healthy, but the depth pipeline becomes silent or returns unusable data.

We would like to understand the intended public-API behavior and the supported way to detect this state without relying on private system logs or an application-defined timeout. We need an AVFoundation-only solution rather than ARKit.

Capture configuration

Our setup follows Apple's TrueDepth streaming sample:

• Discover .builtInTrueDepthCamera.
• Select an activeFormat with a nonempty supportedDepthDataFormats list.
• Set a supported activeDepthDataFormat, typically DepthFloat16.
• Add AVCaptureVideoDataOutput and AVCaptureDepthDataOutput.
• Enable the .depthData connection.
• Test both direct depth delegate delivery and AVCaptureDataOutputSynchronizer.
• Observe capture-session runtime errors, interruptions, connection state, and system pressure.

The same code and configuration continuously deliver valid RGB and depth frames on healthy control devices.

Observed failure modes

1. No depth callback of any kind

• The video delegate continues to receive RGB frames at the expected rate.
• The depth connection reports enabled and active.
• The capture session remains running.
• No AVCaptureSessionRuntimeErrorNotification or interruption is reported.
• System pressure is nominal.
• depthDataOutput(_:didOutput:timestamp:connection:) is never called.
• depthDataOutput(_:didDrop:timestamp:connection:reason:) is also never called.

With AVCaptureDataOutputSynchronizer, RGB synchronization points may continue, but no AVCaptureSynchronizedDepthData object is present for the depth output.

2. One depth frame, followed by permanent silence

On other affected devices, one depth frame is delivered shortly after startup, while hundreds of later RGB frames continue with no further depth output or drop callback.

3. A depth object is delivered, but the map is unusable

We have also observed an AVDepthData object whose map is empty or has no usable spatial information, for example:

• a zero-sized/empty depth representation;
• an unfiltered map containing only NaN; or
• a filtered map containing a single constant value across nearly all pixels.

Metadata such as depth quality, accuracy, calibration availability, supported formats, and connection state does not reliably distinguish these frames from healthy ones.

Relevant documented behavior

The documentation for AVCaptureSynchronizedDepthData.depthDataWasDropped says that a dropped depth object is different from a synchronization timestamp for which no depth capture occurred. In the latter case, no AVCaptureSynchronizedDepthData object is present in the collection.

The documentation for depthDataOutput(_:didDrop:timestamp:connection:reason:) says that the callback reports captured depth data that wasn't processed, with an empty-shell AVDepthData object.

This appears to leave a diagnostic gap: if the TrueDepth pipeline fails before a depth object is produced, the application may receive neither an output callback nor a drop callback.

System-log correlation

We do not want to depend on these private and undocumented strings, but sysdiagnose/device logs from affected devices correlate the public-API silence with messages such as:

Can't activate Pearl projector: no projector token found
PROJECTOR ON DENIED

Object too close (no depth)
GMC status: -2
Projector GMC hasn't completed yet - dropping depth/dx buffers
Pulses exceed limit. Projector Off

In the first group, depth delivery never starts. In the second group, depth may never start or may stop after one frame. RGB capture can remain active in both cases.

These logs suggest that depth or disparity buffers can be rejected before they reach AVCaptureDepthDataOutput, which would explain why the application sees neither didOutput nor didDrop. However, none of these projector, IR, calibration, protection, or token states appears to be exposed by a public API.

Questions

1. Is it expected that AVCaptureDepthDataOutput delivers neither didOutput nor didDrop when no depth object is produced upstream?
2. Is the absence of AVCaptureSynchronizedDepthData from a synchronization collection the only public signal for this condition?
3. Is there a supported API, notification, KVO property, or error that reports that the TrueDepth IR/projector/depth provider is unavailable, disabled, protected, stalled, or unable to produce data?
4. Can an app distinguish a healthy but temporarily delayed depth stream from a pipeline that will never deliver depth, other than by implementing its own timeout and frame-count watchdog?
5. Is there a documented validity rule for an empty, all-NaN, or near-constant AVDepthData.depthDataMap? Should applications treat these as unavailable depth even when the AVDepthData metadata reports a normal quality or accuracy?
6. What is the recommended recovery procedure: rebuild AVCaptureDepthDataOutput, restart the capture session, switch cameras and switch back, wait for a system notification, or stop retrying until the device state changes?
7. ARKit can report ARError.sensorFailed when a required sensor does not deliver input. Is there an AVFoundation equivalent that identifies which required input failed, without adopting ARKit?
8. If no public signal currently exists, would Apple consider exposing a depth-provider status/error callback that distinguishes at least:
   • captured but dropped;
   • no depth capture for the timestamp;
   • provider temporarily unavailable; and
   • provider disabled or failed until recovery/service?

We can provide a minimal sample project, affected-device sysdiagnose, and synchronized timestamp traces in Feedback Assistant if that would help. Please let us know which diagnostics and logging profiles would be most useful.