AVFoundation

RSS for tag

Work with audiovisual assets, control device cameras, process audio, and configure system audio interactions using AVFoundation.

Posts under AVFoundation tag

35 Posts

Post

Replies

Boosts

Views

Activity

Massive CoreML latency spike on live AVFoundation camera feed vs. offline inference (CPU+ANE)
Hello, I’m experiencing a severe performance degradation when running CoreML models on a live AVFoundation video feed compared to offline or synthetic inference. This happens across multiple models I've converted (including SCI, RTMPose, and RTMW) and affects multiple devices. The Environment OS: macOS 26.3, iOS 26.3, iPadOS 26.3 Hardware: Mac14,6 (M2 Max), iPad Pro 11 M1, iPhone 13 mini Compute Units: cpuAndNeuralEngine The Numbers When testing my SCI_output_image_int8.mlpackage model, the inference timings are drastically different: Synthetic/Offline Inference: ~1.34 ms Live Camera Inference: ~15.96 ms Preprocessing is completely ruled out as the bottleneck. My profiling shows total preprocessing (nearest-neighbor resize + feature provider creation) takes only ~0.4 ms in camera mode. Furthermore, no frames are being dropped. What I've Tried I am building a latency-critical app and have implemented almost every recommended optimization to try and fix this, but the camera-feed penalty remains: Matched the AVFoundation camera output format exactly to the model input (640x480 at 30/60fps). Used IOSurface-backed pixel buffers for everything (camera output, synthetic buffer, and resize buffer). Enabled outputBackings. Loaded the model once and reused it for all predictions. Configured MLModelConfiguration with reshapeFrequency = .frequent and specializationStrategy = .fastPrediction. Wrapped inference in ProcessInfo.processInfo.beginActivity(options: .latencyCritical, reason: "CoreML_Inference"). Set DispatchQueue to qos: .userInteractive. Disabled the idle timer and enabled iOS Game Mode. Exported models using coremltools 9.0 (deployment target iOS 26) with ImageType inputs/outputs and INT8 quantization. Reproduction To completely rule out UI or rendering overhead, I wrote a standalone Swift CLI script that isolates the AVFoundation and CoreML pipeline. The script clearly demonstrates the ~15ms latency on live camera frames versus the ~1ms latency on synthetic buffers. (I have attached camera_coreml_benchmark.swift and coreml model (very light low light enghancement model) to this repo on github https://github.com/pzoltowski/apple-coreml-camera-latency-repro). My Question: Is this massive overhead expected behavior for AVFoundation + Core ML on live feeds, or is this a framework/runtime bug? If expected, what is the Apple-recommended pattern to bypass this camera-only inference slowdown? One think found interesting when running in debug model was faster (not as fast as in performance benchmark but faster than 16ms. Also somehow if I did some dummy calculation on on different DispatchQueue also seems like model got slightly faster. So maybe its related to ANE Power State issues (Jitter/SoC Wake) and going to fast to sleep and taking a long time to wakeup? Doing dummy calculation in background thought is probably not a solution. Thanks in advance for any insights!
5
0
1k
Mar ’26
AVAssetDownloadConfiguration: How many video variants are actually downloaded when multiple variants exist in the HLS master playlist?
Hi, I’m trying to better understand how AVAssetDownloadConfiguration selects video variants when downloading HLS content for offline playback. Suppose I have an HLS master playlist (.m3u8) that contains several video variants defined with #EXT-X-STREAM-INF. For example, the master playlist may contain multiple video streams like this: Same resolution, different BANDWIDTH Or different resolutions (for example 720p, 1080p, etc.) My question is: How many video variants are actually downloaded when using AVAssetDownloadConfiguration without specifying any variantQualifiers? In other words: If the master playlist contains multiple video variants, will the download task fetch only one variant, or multiple variants? Does the behavior differ depending on whether the variants differ only by BANDWIDTH or also by RESOLUTION? What I observed in testing In my tests, I always end up with only one video variant downloaded, specifically the one with the highest BANDWIDTH parameter. In the m3u8 files I tested, all video variants had identical parameters (resolution, codec, frame rate, etc.) and differed only by the BANDWIDTH attribute in the master playlist. However, when inspecting the downloaded .movpkg, I noticed something interesting in boot.xml. It lists two video streams: one with complete="true" (the one with highest bandwidth) another with complete="no" (the one with lowest bandwidth) I actually had 3 video streams listed in m3u8, but the one with middle bandwidth wasn't listed in boot.xml file at all. There are also additional streams for audio and subtitles in boot.xml file. This made me wonder whether the system initially attempts to download another video variant (possibly a lower bitrate one), but then switches to the highest-quality variant and only completes that one. Additional question about variantQualifiers If I provide a predicate such as: NSPredicate(format: "peakBitRate > 0") which should theoretically match all variants, will the download task attempt to download all matching video variants, or will it still select only one? Summary So the main questions are: Without variantQualifiers, does AVAssetDownloadConfiguration always download a single video variant, and if so, how is it chosen? Does the behavior differ if variants have different resolutions vs only different bitrates? When a predicate matches multiple variants, can multiple video variants actually be downloaded in a single .movpkg? Why might boot.xml list multiple video streams when only one appears to be fully downloaded? Any clarification on the intended behavior would be greatly appreciated. Thanks!
1
0
400
Mar ’26
10-Bit UVC on iPadOS
Hello, I've been very familiar with the UVC Support in iPadOS ever since it launched in iOS 17. There are a number of people that use the software I've developed built around UVC and there are often queries about 8-Bit vs. 10-Bit. My understanding is that the newest UVC Spec is 1.5 which was standardised in 2012 and almost every UVC Capture Card runs at 8-Bit. The only 10-Bit Capture Card that is on my radar is the AJA U-Tap SDI, however it looks like this is 10-Bit up until the UVC Part where the 10-Bit Input is downsampled to 8-Bit. Though I have read in certain places that it works as a 10-Bit Capture Card on macOS but not on iPadOS. I was just wondering if 10-Bit via UVC is even possible on iPadOS? If there was indeed a true 10-Bit Source being passed into an iPad, would iPadOS allow it or would it be downsampled by AVFoundation so it can show up as a valid external video input? All USB Capture Cards that I have encountered use one of the following formats: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange kCVPixelFormatType_420YpCbCr8BiPlanarFullRange kCVPixelFormatType_32BGRA So if a UVC Device delivered a 10-Bit Format, would that be accessible by iPadOS or would it fallback to these 8-Bit Formats by default? Thanks!
1
0
878
Apr ’26
Lock Screen shows skip buttons instead of next/previous when using MPRemoteCommandCenter with AVPlayer playlist
Hello, I’m building an iOS video player using AVPlayer and a custom playback queue. I implemented remote controls using MPRemoteCommandCenter and enabled: nextTrackCommand previousTrackCommand playCommand pauseCommand I disabled: skipForwardCommand skipBackwardCommand seekForwardCommand seekBackwardCommand I also set queue metadata in MPNowPlayingInfoCenter: MPNowPlayingInfoPropertyPlaybackQueueIndex MPNowPlayingInfoPropertyPlaybackQueueCount Even with these commands enabled and the queue count greater than 1, the iOS lock screen continues to display the 10-second skip buttons instead of the previous/next track buttons. The commands themselves work correctly when triggered externally (Control Center, headphones, etc.), but the UI still shows the skip controls. Is there a way to force the lock screen UI to display previous / next track buttons for a video playlist? Or is this behavior expected when using AVPlayer with video content? Thanks.
0
0
104
Mar ’26
How should playback readiness be determined with AVSampleBufferAudioRenderer when using AirPlay?
I’m implementing a custom playback pipeline using AVSampleBufferAudioRenderer together with AVSampleBufferRenderSynchronizer. hasSufficientMediaDataForReliablePlaybackStart appears to be the intended signal for determining when enough media has been queued to start playback. For local playback, this works well in practice — the property becomes true after a reasonable amount of media is enqueued. However, when the output route is AirPlay, using this property becomes difficult: AirPlay requires significantly more buffered media before the renderer reports sufficient data. The required preroll amount is much larger than for local playback. For short assets, it is possible to enqueue the entire audio track and still never observe hasSufficientMediaDataForReliablePlaybackStart == true. In that situation there is no more media data to enqueue, but the renderer still reports that playback is not ready. Given this behavior, what is the recommended way to determine playback readiness when using AVSampleBufferAudioRenderer with AirPlay?
0
0
421
Mar ’26
Metadata in Video stripped by Share Sheet / Airdrop
I have an application which records video along with some custom metadata and a chapter track. The resultant video is stored in the Camera Roll. When sharing the video via the Share Sheet or AirDrop, the metadata track is stripped entirely (the chapter markers are preserved) Sharing via AirDrop with the "All Photos Data" option does include the metadata track, as does copying from the device with Image Capture but this is a bad user experience as the user must remember to explicitly select this option, and the filename is lost when sending this way. I have also tried various other approaches (such as encoding my metadata in a subtitle track, which I didn't expect to be stripped as it's an accessibility concern) but it's also removed. Essentially I am looking for a definitive list of things that are not stripped or if there's a way to encode a track in some way to indicate it should be preserved. The metadata is added via AVTimedMetadataGroup containing one AVMutableMetadataItem which has its value as a JSON string. I took a different approach with the Chapter Marker track (mainly because I did it first in a completely different way and didn't rework it when I added the other track). I post-process these after the video is recorded, and add them with addMutableTrack and then addTrackAssociation(to: chapterTrack, type: .chapterList) but I don't think that's the reason the chapter track persists where the custom metadata does not as other tests with video files from other sources containing subtitles etc also had their subtitle data stripped. tl;dr I record videos with metadata that I want to be able to share via Share Sheet and AirDrop, what am I doing wrong?
0
0
448
Mar ’26
AVAudioFile.read extremely slow after seeking in FLAC and MP3 files
I'm developing an audio player app that uses AVAudio​File to read PCM data from various formats. I'm experiencing severe performance issues when seeking in FLAC, while other compressed formats (M4A/AAC) work correctly. I don't intend to use them in my app, but I also tested mp3 files just by curiosity and they also have this issue. Environment: macOS 26 (Tahoe) Xcode 26.3 Apple Silicon (M1) The issue: After setting AVAudio​File​.frame​Position to a position mid-file, the subsequent call to AVAudio​File​.read(into​:frame​Count:) blocks for an unreasonable amount of time for FLAC and MP3 files. The delay scales linearly with the seek target, seeking near the beginning is fast, seeking toward the end is proportionally slower, which suggests the decoder is decoding linearly from the beginning of the file rather than using any seek index. (My app deals with “images” of Audio CDs ripped as a single long audio file.) The issue is particularly severe when reading files from an SMB network share (server on Ethernet, client on Wi-Fi with the access point ~2 meters away in line of sight). Quick Benchmark results: I tested with the same 75-minute audio content (16-bit/44.1 kHz stereo, 200,502,708 frames) encoded in five formats, seeking to the midpoint. Over SMB (Local Network, Server on Ethernet, Client on WiFi): Format | Seek + Read Time ----------|------------------ WAV | 0.007 s AIFF | 0.009 s Apple | 0.015 s Lossless | MP3 | 9.2 s FLAC | 30.2 s Locally (MacBook Air M1 SSD) : Format | Seek + Read Time ----------|------------------ WAV | 0.0005 s AIFF | 0.0004 s Apple | 0.0011 s Lossless | MP3 | 0.1958 s FLAC | 0.7528 s WAV, AIFF, and M4A all seek virtually instantly (< 15 ms). MP3 and FLAC exhibit linear-time behavior, with FLAC being the worst affected. Note that M4A (AAC) is also a compressed format that requires decoding after seeking, yet it completes in 15 ms. This rules out any inherent limitation of compressed formats, the MP4 container's packet index (stts/stco) is clearly being used for fast random access. Both MP3 (Xing/LAME TOC) and FLAC (SEEKTABLE metadata block) have their own seek mechanisms that should provide similar performance. Minimal CLI tool to reproduce: import Foundation guard CommandLine.arguments.count > 1 else { print("Usage: FLACSpeed <audio-file-path>") exit(1) } let path = CommandLine.arguments[1] let fileURL = URL(fileURLWithPath: path) do { let file = try AVAudioFile(forReading: fileURL) let format = file.processingFormat let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: 8192)! let totalFrames = file.length let seekTarget = totalFrames / 2 print("File: \(fileURL.lastPathComponent)") print("Format: \(format)") print("Total frames: \(totalFrames)") print("Seeking to frame: \(seekTarget)") file.framePosition = seekTarget let start = CFAbsoluteTimeGetCurrent() try file.read(into: buffer, frameCount: 8192) let elapsed = CFAbsoluteTimeGetCurrent() - start print("Read after seek took \(elapsed) seconds") } catch { print("Error: \(error.localizedDescription)") exit(1) } Expected behavior: AVAudio​File​.read(into​:frame​Count:) after setting frame​Position should use the available seek mechanisms in FLAC and MP3 files for fast random access, as it already does for M4A (AAC). Even accounting for the fact that seek tables provide approximate (not sample-precise) positioning, the "jump to nearest index point + decode forward" approach should complete in milliseconds, not seconds. Workaround: For FLAC, I've worked around this by using libFLAC directly, which provides instant seeking via FLAC__stream​_decoder​_seek​_absolute(). libFLAC Performance: For comparison, libFLAC's FLAC__stream​_decoder​_seek​_absolute() performs the same seek + read on the same FLAC file in around 0.015, using the FLAC seek table to jump to the nearest preceding seek point, then decoding forward a small number of frames to the exact target sample.
1
1
429
Apr ’26
HLS VOD Download Resume after Wifi/LTE Change
Hello there, Our application has HLS VOD download option. Users can download VOD which has DRM protection and watch content when device is offline. We use aggregateAssetDownloadTask to download HLS VOD. We want to resume download after Wifi/LTE change but it does not resume. Download is starting from beginning. We use some token algorithms to start download. That's why, our playlist url and chunks urls may change. But playlist and chunk urls' content is unique. If user start to download via Wifi and changed to LTE. Download request responds 403 Forbidden because of some token algorithms after some time func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) AVAssetDownloadDelegate function triggered by system. If we resume it does not resume at this point. It is starting from beginning if we start new download process for same content. Is there any way to resume unfinished HLS Download processes? Thanks
2
0
566
Mar ’26
Is 18MP Front Camera Capture Available to Third-Party Apps via AVFoundation?
Hi, I'm investigating whether 18MP photo capture from the front camera on iPhone 17 Pro is available to third-party apps using AVFoundation. I first inspected all available AVCaptureDevice formats, but I could not find any format corresponding to ~18MP resolution (e.g., around 4896×3672). for format in device.formats { let desc = format.formatDescription let dims = CMVideoFormatDescriptionGetDimensions(desc) print("Format: (dims.width) x (dims.height)") } All reported formats appear to be limited to resolutions such as 4032×3024 (12MP) or below. Question: Is 18MP front camera capture actually available to third-party apps via AVFoundation on iPhone 17?
1
0
522
Apr ’26
AVPictureInPictureController with AVSampleBufferDisplayLayer: Video not scaled in PiP window on macOS
AVPictureInPictureController with AVSampleBufferDisplayLayer: Video not scaled in PiP window on macOS Platform: macOS 26.4 (Tahoe) Framework: AVKit / AVFoundation Xcode: 26.4 Summary When using AVPictureInPictureController with ContentSource(sampleBufferDisplayLayer:playbackDelegate:) on macOS, the video content in the PiP window is not scaled to fit — it renders at 1:1 pixel resolution, showing only the bottom-left portion of the video (zoomed/cropped). The same code works correctly on iOS. Setup let displayLayer = AVSampleBufferDisplayLayer() displayLayer.videoGravity = .resizeAspect // Host displayLayer as a sublayer of an NSView, enqueue CMSampleBuffers let source = AVPictureInPictureController.ContentSource( sampleBufferDisplayLayer: displayLayer, playbackDelegate: self ) let pip = AVPictureInPictureController(contentSource: source) pip.delegate = self The source display layer is 1280×720, matching the video stream resolution. PiP starts successfully — isPictureInPicturePossible is true, the PiP button works, and the PIPPanel window appears. However, the video in the PiP window (~480×270) shows only the bottom-left 480×270 pixels of the 1280×720 content, rather than scaling the full frame to fit. Investigation Inspecting the PiP window hierarchy reveals: PIPPanel (480×270) └─ AVPictureInPictureSampleBufferDisplayLayerView └─ AVPictureInPictureSampleBufferDisplayLayerHostView (layer = CALayerHost) └─ AVPictureInPictureCALayerHostView The CALayerHost mirrors the source AVSampleBufferDisplayLayer at 1:1 pixel resolution. Unlike AVPlayerLayer-based PiP (which works correctly on macOS), the sample buffer display layer path does not apply any scaling transform to the mirrored content. On iOS, PiP with AVSampleBufferDisplayLayer works correctly because the system reparents the layer into the PiP window, so standard layer scaling applies. On macOS, the system uses CALayerHost mirroring instead, and the scaling step is missing. What I tried (none fix the issue) Setting autoresizingMask on all PiP internal subviews — views resize correctly, but CALayerHost content remains at 1:1 pixel scale Applying CATransform3DMakeScale on the CALayerHost layer — creates a black rectangle artifact; the mirrored content does not transform Setting CALayerHost.bounds to the source layer size — no effect on rendering Reparenting the internal AVPictureInPictureCALayerHostView out of the host view — video disappears entirely Hiding the CALayerHost — PiP window goes white (confirming it is the sole video renderer) Resizing the source AVSampleBufferDisplayLayer to match the PiP window size — partially works (1:1 mirror of a smaller source fits), but causes visible lag during resize, affects the main window's "This video is playing in Picture in Picture" placeholder, and didTransitionToRenderSize stops being called after the initial resize Expected behavior The video content should be scaled to fit the PiP window, respecting the display layer's videoGravity setting (.resizeAspect), consistent with: iOS PiP with AVSampleBufferDisplayLayer (works correctly) macOS PiP with AVPlayerLayer (works correctly) Environment macOS 26.4 (Tahoe) Xcode 26.4 Apple Silicon (M-series) Retina display (contentsScale = 2.0) Video: H.264 1280×720, hardware decoded via VTDecompressionSession, enqueued as CMSampleBuffer
3
0
305
Apr ’26
Solving AVFoundation FigCaptureSourceRemote err=-17281 on iOS 26 — reliable workaround for repeated camera initialization
While working on a heart rate measurement app (photoplethysmography via camera), we faced systematic err=-17281 (FigCaptureSourceRemote) issues on real devices starting from iOS 17+, and the problem became more noticeable after iOS 26. The error often appeared during AVCaptureSession initialization or when restarting capture, especially under high frame rates (30-60 FPS) and frequent foreground/background transitions. Root cause (our understanding): Camera hardware/session not fully released after previous use Race conditions between session teardown and new setup Changes in AVFoundation capture pipeline in recent iOS versions Our solution: Instead of blocking delays, we implemented asynchronous retry logic with explicit hardware readiness check via AVCaptureDevice.lockForConfiguration().
1
0
329
1d
Clarification on WWDC25 Session 300: Do iPhone 11 and SE (2nd gen) fully support Frame Interpolation & Super Resolution without issues?
Hello everyone, I have a question regarding the Ultra-Low Latency Frame Interpolation and Super Resolution features introduced in WWDC 2025 Session 300 (https://developer.apple.com/videos/play/wwdc2025/300/). In the video, it was mentioned that these features run on any device as long as it has iOS 26.0 or later and an Apple Silicon chipset. Based on the official support guide (https://support.apple.com/ko-kr/guide/iphone/iphe3fa5df43/ios), the iPhone 11 and iPhone SE (2nd generation) are listed as supported devices. I just want to double-check and confirm: since they meet the criteria mentioned in the video, do these features actually run without any performance issues or limitations on the iPhone 11 and iPhone SE (2nd gen)? I want to make sure I understand the exact hardware capabilities before proceeding with development. Thanks for your help!
1
0
493
Apr ’26
AVCaptureVideoDataOutput stops zooming while AVCaptureVideoPreviewLayer continues — physical wide / ultra-wide / telephoto only
We use a single AVCaptureSession with AVCaptureVideoPreviewLayer and AVCaptureVideoDataOutput (preview-sized buffers, BGRA). When we increase videoZoomFactor, beyond a certain zoom level the image from AVCaptureVideoDataOutput no longer zooms further, while AVCaptureVideoPreviewLayer continues to zoom with the same zoom control. The preview and the video-data output therefore diverge. This behavior appears when the active camera is a physical lens device — wide, ultra-wide, or telephoto (e.g. builtInWideAngleCamera, builtInUltraWideCamera, builtInTelephotoCamera, or similar). It does not appear when the active input is a virtual / multi-camera (e.g. triple camera, dual-wide, or other system multi-camera). Are there known conditions under which this mismatch between preview and video-data output is expected? Thank you.
1
0
222
Apr ’26
AVContentKeySession: Cannot re-fetch content key once obtained — expected behavior?
We are developing a video streaming app that uses AVContentKeySession with FairPlay Streaming. Our implementation supports both online playback (non-persistable keys) and offline playback (persistable keys). We have observed the following behavior: Once a content key has been obtained for a given Content Key ID, AVContentKeySession does not trigger contentKeySession(_:didProvide:) again for that same Key ID We also attempted to explicitly call processContentKeyRequest(withIdentifier:initializationData:options:) on the session to force a new key request for the same identifier, but this did not result in the delegate callback being fired again. The session appears to consider the key already resolved and silently ignores the request. This means that if a user first plays content online (receiving a non-persistable key), and later wants to download the same content for offline use (requiring a persistable key), the delegate callback is not fired again, and we have no opportunity to request a persistable key. Questions Is this the expected behavior? Specifically, is it by design that AVContentKeySession caches the key for a given Key ID and does not re-request it — even when processContentKeyRequest(withIdentifier:) is explicitly called? Should we use distinct Content Key IDs for persistable vs. non-persistable keys? For example, if the same piece of content can be played both online and offline, is the recommended approach to have the server provide different EXT-X-KEY URIs (and thus different key identifiers) for the streaming and download variants? Is there a supported way to force a fresh key request for a Key ID that has already been resolved — for example, to upgrade from a non-persistable to a persistable key? Environment iOS 18+ AVContentKeySession(keySystem: .fairPlayStreaming) Any guidance on the recommended approach for supporting both streaming and offline playback for the same content would be greatly appreciated.
1
0
474
Apr ’26
iPad Pro M4 giving wrong value for layerPointConverted for ultra wide angle
I am using iPad Pro M4 device to apply exposure point to the camera. While converting layerPointConverted from 0 -1 range to device size point it is giving wrong value. But if same code is used for other iPad like Gen2, it gives proper value. In both cases video gravity used is resizeAspectFill. I tried using true depth camera for M4 device but it does not work.
0
0
232
Apr ’26
iOS 26.4 regression: The `.pauses` audiovisual background playback policy does not pause video playback anymore when backgrounding the app
Starting with iOS 26.4 and the iOS 26.4 SDK, the .pauses audiovisual background playback policy is not correctly applied anymore to an AVPlayer having an attached video layer displayed on screen. This means that, when backgrounding a video-playing app (without Picture in Picture support) or locking the device, playback is not paused automatically by the system anymore. This issue affects the Apple TV application as well. We have filed FB22488151 with more information.
0
0
277
Apr ’26
Setting up video and image capture pipeline creates internal errors in AVFoundation.
I have created code for iOS that allows me to start and stop video acquisition from a proprietary USB camera using AVFoundation's AVCaptureSession and AVCaptureDevice APIs. There is a start and stop method. The start method takes an argument to specify one of two formats that I use for my custom camera application. I can start the session and switch between formats all day without any errors. However, if I start and then stop the camera three times in a row, on the third invocation of start, I get errors in the console output and the CMSampleBuffers stop flowing to my callback. Additionally, once I get AVFoundation into this state, stoping the camera doesn't help. I have to kill the app and start over. Here are the errors. And below these, the code. I'm hoping someone who has experience with these errors or an engineer from Apple who knows the AVFoundation image capture pipeline code, can respond and tell me what I'm doing wrong. Thanks. <<<< FigCaptureSourceRemote >>>> Fig assert: "! storage->connectionDied" at bail (FigCaptureSourceRemote.m:235) - (err=0) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:558) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "! storage->connectionDied" at bail (FigCaptureSourceRemote.m:235) - (err=0) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:253) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:269) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:511) - (err=-16453) Capture session error: The operation could not be completed Capture session error: The operation could not be completed func start(for deviceFormat: String) async throws -> AnyPublisher<CMSampleBuffer, Swift.Error> { func configureCaptureDevice(with deviceFormat: String) throws { guard let format = formatDict[deviceFormat] else { throw Error.captureFormatNotFound } captureSession.beginConfiguration() defer { captureSession.commitConfiguration() } try captureDevice.lockForConfiguration() captureDeviceFormat = deviceFormat captureDevice.activeFormat = format captureDevice.unlockForConfiguration() } return try await withCheckedThrowingContinuation { continuation in sessionQueue.async { [unowned self] in logger.debug("Start capture session for \(deviceFormat): \(String(describing: captureSession))") // If we were already steaming camera images from a different mode, terminate that stream. bufferPublisher?.send(completion: .finished) bufferPublisher = nil captureDeviceFormat = "" do { // Re-configure with the new format; should be harmless if called with the currently configured format. try configureCaptureDevice(with: deviceFormat) // Return a new stream publisher for this invocation. bufferPublisher = PassthroughSubject<CMSampleBuffer, Swift.Error>() // If we are not currently running, start the image capture pipeline. if captureSession.isRunning == false { captureSession.startRunning() } continuation.resume(returning: bufferPublisher!.eraseToAnyPublisher()) } catch { logger.fault("Failed to start camera: \(error.localizedDescription)") continuation.resume(throwing: error) } } } } func stop() async throws { try await withCheckedThrowingContinuation { continuation in sessionQueue.async { [unowned self] in logger.debug("Stop capture session: \(String(describing: captureSession))") // The following invocation is synchronous and takes time to execute; // looks like a stall but you can ignore it as the MainActor is not blocked. captureSession.stopRunning() // Terminate the stream and reset our state. bufferPublisher?.send(completion: .finished) bufferPublisher = nil captureDeviceFormat = "" // Signal the caller that we are done here. continuation.resume() } } }
0
0
274
Apr ’26
Massive CoreML latency spike on live AVFoundation camera feed vs. offline inference (CPU+ANE)
Hello, I’m experiencing a severe performance degradation when running CoreML models on a live AVFoundation video feed compared to offline or synthetic inference. This happens across multiple models I've converted (including SCI, RTMPose, and RTMW) and affects multiple devices. The Environment OS: macOS 26.3, iOS 26.3, iPadOS 26.3 Hardware: Mac14,6 (M2 Max), iPad Pro 11 M1, iPhone 13 mini Compute Units: cpuAndNeuralEngine The Numbers When testing my SCI_output_image_int8.mlpackage model, the inference timings are drastically different: Synthetic/Offline Inference: ~1.34 ms Live Camera Inference: ~15.96 ms Preprocessing is completely ruled out as the bottleneck. My profiling shows total preprocessing (nearest-neighbor resize + feature provider creation) takes only ~0.4 ms in camera mode. Furthermore, no frames are being dropped. What I've Tried I am building a latency-critical app and have implemented almost every recommended optimization to try and fix this, but the camera-feed penalty remains: Matched the AVFoundation camera output format exactly to the model input (640x480 at 30/60fps). Used IOSurface-backed pixel buffers for everything (camera output, synthetic buffer, and resize buffer). Enabled outputBackings. Loaded the model once and reused it for all predictions. Configured MLModelConfiguration with reshapeFrequency = .frequent and specializationStrategy = .fastPrediction. Wrapped inference in ProcessInfo.processInfo.beginActivity(options: .latencyCritical, reason: "CoreML_Inference"). Set DispatchQueue to qos: .userInteractive. Disabled the idle timer and enabled iOS Game Mode. Exported models using coremltools 9.0 (deployment target iOS 26) with ImageType inputs/outputs and INT8 quantization. Reproduction To completely rule out UI or rendering overhead, I wrote a standalone Swift CLI script that isolates the AVFoundation and CoreML pipeline. The script clearly demonstrates the ~15ms latency on live camera frames versus the ~1ms latency on synthetic buffers. (I have attached camera_coreml_benchmark.swift and coreml model (very light low light enghancement model) to this repo on github https://github.com/pzoltowski/apple-coreml-camera-latency-repro). My Question: Is this massive overhead expected behavior for AVFoundation + Core ML on live feeds, or is this a framework/runtime bug? If expected, what is the Apple-recommended pattern to bypass this camera-only inference slowdown? One think found interesting when running in debug model was faster (not as fast as in performance benchmark but faster than 16ms. Also somehow if I did some dummy calculation on on different DispatchQueue also seems like model got slightly faster. So maybe its related to ANE Power State issues (Jitter/SoC Wake) and going to fast to sleep and taking a long time to wakeup? Doing dummy calculation in background thought is probably not a solution. Thanks in advance for any insights!
Replies
5
Boosts
0
Views
1k
Activity
Mar ’26
AVAssetDownloadConfiguration: How many video variants are actually downloaded when multiple variants exist in the HLS master playlist?
Hi, I’m trying to better understand how AVAssetDownloadConfiguration selects video variants when downloading HLS content for offline playback. Suppose I have an HLS master playlist (.m3u8) that contains several video variants defined with #EXT-X-STREAM-INF. For example, the master playlist may contain multiple video streams like this: Same resolution, different BANDWIDTH Or different resolutions (for example 720p, 1080p, etc.) My question is: How many video variants are actually downloaded when using AVAssetDownloadConfiguration without specifying any variantQualifiers? In other words: If the master playlist contains multiple video variants, will the download task fetch only one variant, or multiple variants? Does the behavior differ depending on whether the variants differ only by BANDWIDTH or also by RESOLUTION? What I observed in testing In my tests, I always end up with only one video variant downloaded, specifically the one with the highest BANDWIDTH parameter. In the m3u8 files I tested, all video variants had identical parameters (resolution, codec, frame rate, etc.) and differed only by the BANDWIDTH attribute in the master playlist. However, when inspecting the downloaded .movpkg, I noticed something interesting in boot.xml. It lists two video streams: one with complete="true" (the one with highest bandwidth) another with complete="no" (the one with lowest bandwidth) I actually had 3 video streams listed in m3u8, but the one with middle bandwidth wasn't listed in boot.xml file at all. There are also additional streams for audio and subtitles in boot.xml file. This made me wonder whether the system initially attempts to download another video variant (possibly a lower bitrate one), but then switches to the highest-quality variant and only completes that one. Additional question about variantQualifiers If I provide a predicate such as: NSPredicate(format: "peakBitRate > 0") which should theoretically match all variants, will the download task attempt to download all matching video variants, or will it still select only one? Summary So the main questions are: Without variantQualifiers, does AVAssetDownloadConfiguration always download a single video variant, and if so, how is it chosen? Does the behavior differ if variants have different resolutions vs only different bitrates? When a predicate matches multiple variants, can multiple video variants actually be downloaded in a single .movpkg? Why might boot.xml list multiple video streams when only one appears to be fully downloaded? Any clarification on the intended behavior would be greatly appreciated. Thanks!
Replies
1
Boosts
0
Views
400
Activity
Mar ’26
Audio System Trace: Zero Time Stamp
In Instruments, I'm seeing "Zero Time Stamp" events in the "Audio Server" lane. What does that mean?
Replies
1
Boosts
0
Views
218
Activity
Mar ’26
10-Bit UVC on iPadOS
Hello, I've been very familiar with the UVC Support in iPadOS ever since it launched in iOS 17. There are a number of people that use the software I've developed built around UVC and there are often queries about 8-Bit vs. 10-Bit. My understanding is that the newest UVC Spec is 1.5 which was standardised in 2012 and almost every UVC Capture Card runs at 8-Bit. The only 10-Bit Capture Card that is on my radar is the AJA U-Tap SDI, however it looks like this is 10-Bit up until the UVC Part where the 10-Bit Input is downsampled to 8-Bit. Though I have read in certain places that it works as a 10-Bit Capture Card on macOS but not on iPadOS. I was just wondering if 10-Bit via UVC is even possible on iPadOS? If there was indeed a true 10-Bit Source being passed into an iPad, would iPadOS allow it or would it be downsampled by AVFoundation so it can show up as a valid external video input? All USB Capture Cards that I have encountered use one of the following formats: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange kCVPixelFormatType_420YpCbCr8BiPlanarFullRange kCVPixelFormatType_32BGRA So if a UVC Device delivered a 10-Bit Format, would that be accessible by iPadOS or would it fallback to these 8-Bit Formats by default? Thanks!
Replies
1
Boosts
0
Views
878
Activity
Apr ’26
Lock Screen shows skip buttons instead of next/previous when using MPRemoteCommandCenter with AVPlayer playlist
Hello, I’m building an iOS video player using AVPlayer and a custom playback queue. I implemented remote controls using MPRemoteCommandCenter and enabled: nextTrackCommand previousTrackCommand playCommand pauseCommand I disabled: skipForwardCommand skipBackwardCommand seekForwardCommand seekBackwardCommand I also set queue metadata in MPNowPlayingInfoCenter: MPNowPlayingInfoPropertyPlaybackQueueIndex MPNowPlayingInfoPropertyPlaybackQueueCount Even with these commands enabled and the queue count greater than 1, the iOS lock screen continues to display the 10-second skip buttons instead of the previous/next track buttons. The commands themselves work correctly when triggered externally (Control Center, headphones, etc.), but the UI still shows the skip controls. Is there a way to force the lock screen UI to display previous / next track buttons for a video playlist? Or is this behavior expected when using AVPlayer with video content? Thanks.
Replies
0
Boosts
0
Views
104
Activity
Mar ’26
How should playback readiness be determined with AVSampleBufferAudioRenderer when using AirPlay?
I’m implementing a custom playback pipeline using AVSampleBufferAudioRenderer together with AVSampleBufferRenderSynchronizer. hasSufficientMediaDataForReliablePlaybackStart appears to be the intended signal for determining when enough media has been queued to start playback. For local playback, this works well in practice — the property becomes true after a reasonable amount of media is enqueued. However, when the output route is AirPlay, using this property becomes difficult: AirPlay requires significantly more buffered media before the renderer reports sufficient data. The required preroll amount is much larger than for local playback. For short assets, it is possible to enqueue the entire audio track and still never observe hasSufficientMediaDataForReliablePlaybackStart == true. In that situation there is no more media data to enqueue, but the renderer still reports that playback is not ready. Given this behavior, what is the recommended way to determine playback readiness when using AVSampleBufferAudioRenderer with AirPlay?
Replies
0
Boosts
0
Views
421
Activity
Mar ’26
Metadata in Video stripped by Share Sheet / Airdrop
I have an application which records video along with some custom metadata and a chapter track. The resultant video is stored in the Camera Roll. When sharing the video via the Share Sheet or AirDrop, the metadata track is stripped entirely (the chapter markers are preserved) Sharing via AirDrop with the "All Photos Data" option does include the metadata track, as does copying from the device with Image Capture but this is a bad user experience as the user must remember to explicitly select this option, and the filename is lost when sending this way. I have also tried various other approaches (such as encoding my metadata in a subtitle track, which I didn't expect to be stripped as it's an accessibility concern) but it's also removed. Essentially I am looking for a definitive list of things that are not stripped or if there's a way to encode a track in some way to indicate it should be preserved. The metadata is added via AVTimedMetadataGroup containing one AVMutableMetadataItem which has its value as a JSON string. I took a different approach with the Chapter Marker track (mainly because I did it first in a completely different way and didn't rework it when I added the other track). I post-process these after the video is recorded, and add them with addMutableTrack and then addTrackAssociation(to: chapterTrack, type: .chapterList) but I don't think that's the reason the chapter track persists where the custom metadata does not as other tests with video files from other sources containing subtitles etc also had their subtitle data stripped. tl;dr I record videos with metadata that I want to be able to share via Share Sheet and AirDrop, what am I doing wrong?
Replies
0
Boosts
0
Views
448
Activity
Mar ’26
AVAudioFile.read extremely slow after seeking in FLAC and MP3 files
I'm developing an audio player app that uses AVAudio​File to read PCM data from various formats. I'm experiencing severe performance issues when seeking in FLAC, while other compressed formats (M4A/AAC) work correctly. I don't intend to use them in my app, but I also tested mp3 files just by curiosity and they also have this issue. Environment: macOS 26 (Tahoe) Xcode 26.3 Apple Silicon (M1) The issue: After setting AVAudio​File​.frame​Position to a position mid-file, the subsequent call to AVAudio​File​.read(into​:frame​Count:) blocks for an unreasonable amount of time for FLAC and MP3 files. The delay scales linearly with the seek target, seeking near the beginning is fast, seeking toward the end is proportionally slower, which suggests the decoder is decoding linearly from the beginning of the file rather than using any seek index. (My app deals with “images” of Audio CDs ripped as a single long audio file.) The issue is particularly severe when reading files from an SMB network share (server on Ethernet, client on Wi-Fi with the access point ~2 meters away in line of sight). Quick Benchmark results: I tested with the same 75-minute audio content (16-bit/44.1 kHz stereo, 200,502,708 frames) encoded in five formats, seeking to the midpoint. Over SMB (Local Network, Server on Ethernet, Client on WiFi): Format | Seek + Read Time ----------|------------------ WAV | 0.007 s AIFF | 0.009 s Apple | 0.015 s Lossless | MP3 | 9.2 s FLAC | 30.2 s Locally (MacBook Air M1 SSD) : Format | Seek + Read Time ----------|------------------ WAV | 0.0005 s AIFF | 0.0004 s Apple | 0.0011 s Lossless | MP3 | 0.1958 s FLAC | 0.7528 s WAV, AIFF, and M4A all seek virtually instantly (< 15 ms). MP3 and FLAC exhibit linear-time behavior, with FLAC being the worst affected. Note that M4A (AAC) is also a compressed format that requires decoding after seeking, yet it completes in 15 ms. This rules out any inherent limitation of compressed formats, the MP4 container's packet index (stts/stco) is clearly being used for fast random access. Both MP3 (Xing/LAME TOC) and FLAC (SEEKTABLE metadata block) have their own seek mechanisms that should provide similar performance. Minimal CLI tool to reproduce: import Foundation guard CommandLine.arguments.count > 1 else { print("Usage: FLACSpeed <audio-file-path>") exit(1) } let path = CommandLine.arguments[1] let fileURL = URL(fileURLWithPath: path) do { let file = try AVAudioFile(forReading: fileURL) let format = file.processingFormat let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: 8192)! let totalFrames = file.length let seekTarget = totalFrames / 2 print("File: \(fileURL.lastPathComponent)") print("Format: \(format)") print("Total frames: \(totalFrames)") print("Seeking to frame: \(seekTarget)") file.framePosition = seekTarget let start = CFAbsoluteTimeGetCurrent() try file.read(into: buffer, frameCount: 8192) let elapsed = CFAbsoluteTimeGetCurrent() - start print("Read after seek took \(elapsed) seconds") } catch { print("Error: \(error.localizedDescription)") exit(1) } Expected behavior: AVAudio​File​.read(into​:frame​Count:) after setting frame​Position should use the available seek mechanisms in FLAC and MP3 files for fast random access, as it already does for M4A (AAC). Even accounting for the fact that seek tables provide approximate (not sample-precise) positioning, the "jump to nearest index point + decode forward" approach should complete in milliseconds, not seconds. Workaround: For FLAC, I've worked around this by using libFLAC directly, which provides instant seeking via FLAC__stream​_decoder​_seek​_absolute(). libFLAC Performance: For comparison, libFLAC's FLAC__stream​_decoder​_seek​_absolute() performs the same seek + read on the same FLAC file in around 0.015, using the FLAC seek table to jump to the nearest preceding seek point, then decoding forward a small number of frames to the exact target sample.
Replies
1
Boosts
1
Views
429
Activity
Apr ’26
HLS VOD Download Resume after Wifi/LTE Change
Hello there, Our application has HLS VOD download option. Users can download VOD which has DRM protection and watch content when device is offline. We use aggregateAssetDownloadTask to download HLS VOD. We want to resume download after Wifi/LTE change but it does not resume. Download is starting from beginning. We use some token algorithms to start download. That's why, our playlist url and chunks urls may change. But playlist and chunk urls' content is unique. If user start to download via Wifi and changed to LTE. Download request responds 403 Forbidden because of some token algorithms after some time func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) AVAssetDownloadDelegate function triggered by system. If we resume it does not resume at this point. It is starting from beginning if we start new download process for same content. Is there any way to resume unfinished HLS Download processes? Thanks
Replies
2
Boosts
0
Views
566
Activity
Mar ’26
After upgrade to iOS 26.4, averagePowerLevel and peakHoldLevel are stuck -120
We have an application that capture audio and video. App captures audio PCM on internal or external microphone and displays audio level on the screen. App was working fine for many years but after iOS 26.4 upgrade, averagePowerLevel and peakHoldLevel are stuck to -120 values. Any suggestion?
Replies
7
Boosts
5
Views
1.1k
Activity
1w
How to hide route button `showsRouteButton = false` in `MPVolumeView` without deprecation warning?
MPVolumeView's showsRouteButton was deprecated (https://developer.apple.com/documentation/mediaplayer/mpvolumeview/showsroutebutton?language=objc). It's not clear how can we now hide this button without deprecation warning. The documentation is lacking. Please advise. Thank you!
Replies
4
Boosts
0
Views
414
Activity
Mar ’26
Is 18MP Front Camera Capture Available to Third-Party Apps via AVFoundation?
Hi, I'm investigating whether 18MP photo capture from the front camera on iPhone 17 Pro is available to third-party apps using AVFoundation. I first inspected all available AVCaptureDevice formats, but I could not find any format corresponding to ~18MP resolution (e.g., around 4896×3672). for format in device.formats { let desc = format.formatDescription let dims = CMVideoFormatDescriptionGetDimensions(desc) print("Format: (dims.width) x (dims.height)") } All reported formats appear to be limited to resolutions such as 4032×3024 (12MP) or below. Question: Is 18MP front camera capture actually available to third-party apps via AVFoundation on iPhone 17?
Replies
1
Boosts
0
Views
522
Activity
Apr ’26
AVPictureInPictureController with AVSampleBufferDisplayLayer: Video not scaled in PiP window on macOS
AVPictureInPictureController with AVSampleBufferDisplayLayer: Video not scaled in PiP window on macOS Platform: macOS 26.4 (Tahoe) Framework: AVKit / AVFoundation Xcode: 26.4 Summary When using AVPictureInPictureController with ContentSource(sampleBufferDisplayLayer:playbackDelegate:) on macOS, the video content in the PiP window is not scaled to fit — it renders at 1:1 pixel resolution, showing only the bottom-left portion of the video (zoomed/cropped). The same code works correctly on iOS. Setup let displayLayer = AVSampleBufferDisplayLayer() displayLayer.videoGravity = .resizeAspect // Host displayLayer as a sublayer of an NSView, enqueue CMSampleBuffers let source = AVPictureInPictureController.ContentSource( sampleBufferDisplayLayer: displayLayer, playbackDelegate: self ) let pip = AVPictureInPictureController(contentSource: source) pip.delegate = self The source display layer is 1280×720, matching the video stream resolution. PiP starts successfully — isPictureInPicturePossible is true, the PiP button works, and the PIPPanel window appears. However, the video in the PiP window (~480×270) shows only the bottom-left 480×270 pixels of the 1280×720 content, rather than scaling the full frame to fit. Investigation Inspecting the PiP window hierarchy reveals: PIPPanel (480×270) └─ AVPictureInPictureSampleBufferDisplayLayerView └─ AVPictureInPictureSampleBufferDisplayLayerHostView (layer = CALayerHost) └─ AVPictureInPictureCALayerHostView The CALayerHost mirrors the source AVSampleBufferDisplayLayer at 1:1 pixel resolution. Unlike AVPlayerLayer-based PiP (which works correctly on macOS), the sample buffer display layer path does not apply any scaling transform to the mirrored content. On iOS, PiP with AVSampleBufferDisplayLayer works correctly because the system reparents the layer into the PiP window, so standard layer scaling applies. On macOS, the system uses CALayerHost mirroring instead, and the scaling step is missing. What I tried (none fix the issue) Setting autoresizingMask on all PiP internal subviews — views resize correctly, but CALayerHost content remains at 1:1 pixel scale Applying CATransform3DMakeScale on the CALayerHost layer — creates a black rectangle artifact; the mirrored content does not transform Setting CALayerHost.bounds to the source layer size — no effect on rendering Reparenting the internal AVPictureInPictureCALayerHostView out of the host view — video disappears entirely Hiding the CALayerHost — PiP window goes white (confirming it is the sole video renderer) Resizing the source AVSampleBufferDisplayLayer to match the PiP window size — partially works (1:1 mirror of a smaller source fits), but causes visible lag during resize, affects the main window's "This video is playing in Picture in Picture" placeholder, and didTransitionToRenderSize stops being called after the initial resize Expected behavior The video content should be scaled to fit the PiP window, respecting the display layer's videoGravity setting (.resizeAspect), consistent with: iOS PiP with AVSampleBufferDisplayLayer (works correctly) macOS PiP with AVPlayerLayer (works correctly) Environment macOS 26.4 (Tahoe) Xcode 26.4 Apple Silicon (M-series) Retina display (contentsScale = 2.0) Video: H.264 1280×720, hardware decoded via VTDecompressionSession, enqueued as CMSampleBuffer
Replies
3
Boosts
0
Views
305
Activity
Apr ’26
Solving AVFoundation FigCaptureSourceRemote err=-17281 on iOS 26 — reliable workaround for repeated camera initialization
While working on a heart rate measurement app (photoplethysmography via camera), we faced systematic err=-17281 (FigCaptureSourceRemote) issues on real devices starting from iOS 17+, and the problem became more noticeable after iOS 26. The error often appeared during AVCaptureSession initialization or when restarting capture, especially under high frame rates (30-60 FPS) and frequent foreground/background transitions. Root cause (our understanding): Camera hardware/session not fully released after previous use Race conditions between session teardown and new setup Changes in AVFoundation capture pipeline in recent iOS versions Our solution: Instead of blocking delays, we implemented asynchronous retry logic with explicit hardware readiness check via AVCaptureDevice.lockForConfiguration().
Replies
1
Boosts
0
Views
329
Activity
1d
Clarification on WWDC25 Session 300: Do iPhone 11 and SE (2nd gen) fully support Frame Interpolation & Super Resolution without issues?
Hello everyone, I have a question regarding the Ultra-Low Latency Frame Interpolation and Super Resolution features introduced in WWDC 2025 Session 300 (https://developer.apple.com/videos/play/wwdc2025/300/). In the video, it was mentioned that these features run on any device as long as it has iOS 26.0 or later and an Apple Silicon chipset. Based on the official support guide (https://support.apple.com/ko-kr/guide/iphone/iphe3fa5df43/ios), the iPhone 11 and iPhone SE (2nd generation) are listed as supported devices. I just want to double-check and confirm: since they meet the criteria mentioned in the video, do these features actually run without any performance issues or limitations on the iPhone 11 and iPhone SE (2nd gen)? I want to make sure I understand the exact hardware capabilities before proceeding with development. Thanks for your help!
Replies
1
Boosts
0
Views
493
Activity
Apr ’26
AVCaptureVideoDataOutput stops zooming while AVCaptureVideoPreviewLayer continues — physical wide / ultra-wide / telephoto only
We use a single AVCaptureSession with AVCaptureVideoPreviewLayer and AVCaptureVideoDataOutput (preview-sized buffers, BGRA). When we increase videoZoomFactor, beyond a certain zoom level the image from AVCaptureVideoDataOutput no longer zooms further, while AVCaptureVideoPreviewLayer continues to zoom with the same zoom control. The preview and the video-data output therefore diverge. This behavior appears when the active camera is a physical lens device — wide, ultra-wide, or telephoto (e.g. builtInWideAngleCamera, builtInUltraWideCamera, builtInTelephotoCamera, or similar). It does not appear when the active input is a virtual / multi-camera (e.g. triple camera, dual-wide, or other system multi-camera). Are there known conditions under which this mismatch between preview and video-data output is expected? Thank you.
Replies
1
Boosts
0
Views
222
Activity
Apr ’26
AVContentKeySession: Cannot re-fetch content key once obtained — expected behavior?
We are developing a video streaming app that uses AVContentKeySession with FairPlay Streaming. Our implementation supports both online playback (non-persistable keys) and offline playback (persistable keys). We have observed the following behavior: Once a content key has been obtained for a given Content Key ID, AVContentKeySession does not trigger contentKeySession(_:didProvide:) again for that same Key ID We also attempted to explicitly call processContentKeyRequest(withIdentifier:initializationData:options:) on the session to force a new key request for the same identifier, but this did not result in the delegate callback being fired again. The session appears to consider the key already resolved and silently ignores the request. This means that if a user first plays content online (receiving a non-persistable key), and later wants to download the same content for offline use (requiring a persistable key), the delegate callback is not fired again, and we have no opportunity to request a persistable key. Questions Is this the expected behavior? Specifically, is it by design that AVContentKeySession caches the key for a given Key ID and does not re-request it — even when processContentKeyRequest(withIdentifier:) is explicitly called? Should we use distinct Content Key IDs for persistable vs. non-persistable keys? For example, if the same piece of content can be played both online and offline, is the recommended approach to have the server provide different EXT-X-KEY URIs (and thus different key identifiers) for the streaming and download variants? Is there a supported way to force a fresh key request for a Key ID that has already been resolved — for example, to upgrade from a non-persistable to a persistable key? Environment iOS 18+ AVContentKeySession(keySystem: .fairPlayStreaming) Any guidance on the recommended approach for supporting both streaming and offline playback for the same content would be greatly appreciated.
Replies
1
Boosts
0
Views
474
Activity
Apr ’26
iPad Pro M4 giving wrong value for layerPointConverted for ultra wide angle
I am using iPad Pro M4 device to apply exposure point to the camera. While converting layerPointConverted from 0 -1 range to device size point it is giving wrong value. But if same code is used for other iPad like Gen2, it gives proper value. In both cases video gravity used is resizeAspectFill. I tried using true depth camera for M4 device but it does not work.
Replies
0
Boosts
0
Views
232
Activity
Apr ’26
iOS 26.4 regression: The `.pauses` audiovisual background playback policy does not pause video playback anymore when backgrounding the app
Starting with iOS 26.4 and the iOS 26.4 SDK, the .pauses audiovisual background playback policy is not correctly applied anymore to an AVPlayer having an attached video layer displayed on screen. This means that, when backgrounding a video-playing app (without Picture in Picture support) or locking the device, playback is not paused automatically by the system anymore. This issue affects the Apple TV application as well. We have filed FB22488151 with more information.
Replies
0
Boosts
0
Views
277
Activity
Apr ’26
Setting up video and image capture pipeline creates internal errors in AVFoundation.
I have created code for iOS that allows me to start and stop video acquisition from a proprietary USB camera using AVFoundation's AVCaptureSession and AVCaptureDevice APIs. There is a start and stop method. The start method takes an argument to specify one of two formats that I use for my custom camera application. I can start the session and switch between formats all day without any errors. However, if I start and then stop the camera three times in a row, on the third invocation of start, I get errors in the console output and the CMSampleBuffers stop flowing to my callback. Additionally, once I get AVFoundation into this state, stoping the camera doesn't help. I have to kill the app and start over. Here are the errors. And below these, the code. I'm hoping someone who has experience with these errors or an engineer from Apple who knows the AVFoundation image capture pipeline code, can respond and tell me what I'm doing wrong. Thanks. <<<< FigCaptureSourceRemote >>>> Fig assert: "! storage->connectionDied" at bail (FigCaptureSourceRemote.m:235) - (err=0) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:558) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "! storage->connectionDied" at bail (FigCaptureSourceRemote.m:235) - (err=0) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:253) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:269) - (err=-16453) <<<< FigCaptureSourceRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSourceRemote.m:511) - (err=-16453) Capture session error: The operation could not be completed Capture session error: The operation could not be completed func start(for deviceFormat: String) async throws -> AnyPublisher<CMSampleBuffer, Swift.Error> { func configureCaptureDevice(with deviceFormat: String) throws { guard let format = formatDict[deviceFormat] else { throw Error.captureFormatNotFound } captureSession.beginConfiguration() defer { captureSession.commitConfiguration() } try captureDevice.lockForConfiguration() captureDeviceFormat = deviceFormat captureDevice.activeFormat = format captureDevice.unlockForConfiguration() } return try await withCheckedThrowingContinuation { continuation in sessionQueue.async { [unowned self] in logger.debug("Start capture session for \(deviceFormat): \(String(describing: captureSession))") // If we were already steaming camera images from a different mode, terminate that stream. bufferPublisher?.send(completion: .finished) bufferPublisher = nil captureDeviceFormat = "" do { // Re-configure with the new format; should be harmless if called with the currently configured format. try configureCaptureDevice(with: deviceFormat) // Return a new stream publisher for this invocation. bufferPublisher = PassthroughSubject<CMSampleBuffer, Swift.Error>() // If we are not currently running, start the image capture pipeline. if captureSession.isRunning == false { captureSession.startRunning() } continuation.resume(returning: bufferPublisher!.eraseToAnyPublisher()) } catch { logger.fault("Failed to start camera: \(error.localizedDescription)") continuation.resume(throwing: error) } } } } func stop() async throws { try await withCheckedThrowingContinuation { continuation in sessionQueue.async { [unowned self] in logger.debug("Stop capture session: \(String(describing: captureSession))") // The following invocation is synchronous and takes time to execute; // looks like a stall but you can ignore it as the MainActor is not blocked. captureSession.stopRunning() // Terminate the stream and reset our state. bufferPublisher?.send(completion: .finished) bufferPublisher = nil captureDeviceFormat = "" // Signal the caller that we are done here. continuation.resume() } } }
Replies
0
Boosts
0
Views
274
Activity
Apr ’26