AVFoundation

RSS for tag

Work with audiovisual assets, control device cameras, process audio, and configure system audio interactions using AVFoundation.

Posts under AVFoundation tag

200 Posts

Post

Replies

Boosts

Views

Activity

CallKit VoIP call on the receiver: the side (lock) button only locks the screen, no CXEndCallAction is delivered
Device and versions iPhone 15, iOS 27.0.1 App: Hush, a private messenger. Capacitor 8.5.2 iOS app; the UI is a web page in a WKWebView. Call signalling runs in the page; call audio runs in a native Swift Capacitor plugin on stasel/WebRTC 154.0.0, integrated with CallKit. Built on Codemagic with the latest Xcode, distributed through TestFlight. UIBackgroundModes: audio, voip. No PushKit yet: calls are placed and answered in the app's own screen while it is in the foreground. Setup RTCAudioSession manual audio (useManualAudio = true). isAudioEnabled is set only after ICE connects AND provider(_:didActivate:) is called (a 3-second fallback exists but never fired in these tests). We call RTCAudioSession.audioSessionDidActivate / audioSessionDidDeactivate from the provider delegate. Session: playAndRecord, mode voiceChat, options allowBluetoothHFP, no defaultToSpeaker. overrideOutputAudioPort is only called when the user picks Speaker (never in the failing tests). One CXProvider, created lazily, delegate on the main queue. Configuration: supportsVideo false (also tested true), maximumCallGroups 1, maximumCallsPerCallGroup 1, supportedHandleTypes [.generic], includesCallsInRecents false (also tested true). Outgoing: CXStartCallAction is requested via CXCallController when ICE first connects (not at dial time; the page plays its own ringback until then), handle generic, value "Hush". In perform(CXStartCallAction) we configure the audio session (no activation), reportCall(with:updated:), fulfil, then reportOutgoingCall(startedConnectingAt:) and (connectedAt:). Answering side: reportNewIncomingCall, then an immediate CXAnswerCallAction (the user already accepted in the app's screen). Mute: CXSetMutedCallAction. In-app End reported the call with reportCall(with:endedAt:reason: .remoteEnded) in all tests below (we are changing this to a CXEndCallAction request; not yet tested). Proximity: isProximityMonitoringEnabled = true from dialling while on the receiver. The screen does go dark at the ear. Also tested with our proximity monitoring off. The problem With a Hush call live, route Receiver, CallKit holding the call (the system call UI shows it), didActivate delivered exactly once, our audio running inside that activation, proximity on and the screen dark at the ear: pressing the side button only locks the screen. No CXEndCallAction arrives and the call continues. In the Phone app on the same iPhone the side button ends the call at the ear. Snapchat calls on this iPhone end on the side button while Snapchat's call screen is open. What works: End on the system lock-screen call UI delivers a CXEndCallAction and we end the call. Mute from the system UI arrives as CXSetMutedCallAction. So provider delegate delivery works. What we tried, in order (each tested on device, with a readout of the plugin's state after the call) First CallKit build: side button did nothing; lock-screen End worked. Readout: "CallKit off, its audio off (0x), fallback no, route speaker, proximity on". Cause: CXStartCallAction was refused (missing voip background mode) and the error swallowed. Fixed: voip added; on CXErrorCodeRequestTransactionError.unknownCallProvider we invalidate and recreate the provider and retry once. Next build: CallKit held the call, didActivate once, route Receiver. Side button still did nothing, no CXEndCallAction. Our route log showed two route changes caused by our own overrideOutputAudioPort(.none) on every activation and route change. Removed: no override unless Speaker is selected. Result: no change. Diagnostic switches, tested in combinations: our proximity monitoring on/off; supportsVideo false/true; includesCallsInRecents false/true; callee now reported with reportNewIncomingCall + CXAnswerCallAction (previously both sides used CXStartCallAction). Result: the side button ended the call EXACTLY ONCE: the first call after the provider was recreated (changing supportsVideo makes us invalidate() and create a new CXProvider). Every later call with the same configuration failed. Readout for the failures: "CallKit on, its audio on (1x), fallback no, route Receiver, proximity on", screen dark at the ear. Following thread 837211 (creating a provider implicitly pushes the audio session ID to callservicesd, which can keep a stale one; set the configuration before reporting each call): we now assign provider.configuration = a fresh CXProviderConfiguration at call start and again immediately before each CXStartCallAction request / reportNewIncomingCall. We also added a switch that recreates the CXProvider for every call. With the re-push alone: no end action. With a fresh provider every call: no end action either. Readout each time: "provider calls 1, config re-pushed yes, CallKit on, its audio on (1x), route Receiver, proximity on". The single success in step 3 has not been reproduced. Not tried on purpose: detecting the lock ourselves (protectedDataWillBecomeUnavailable arrives about 10 seconds late and needs a passcode; a 2017 thread says Home and Lock cannot be told apart); reporting the CallKit call at dial time instead of ICE connect; PushKit. Threads we relied on 819363 (April 2026): the lock button hangs up a CallKit call handled directly by the device (not through a headset), delivered as the provider's end action. 837211 (August 2026): stale audio session ID in callservicesd after provider creation; set the configuration before reporting a call. How we differ from apps where it works WKWebView: WebKit's GPU process holds a second AVAudioSession in our app's name. During the call the page captures no mic (the plugin owns it); before connect the page plays a Web Audio ringback (Ambient category). The CallKit call starts at ICE connect, not at dial time. In-app End reported as .remoteEnded rather than requested as a CXEndCallAction. The callee's immediate CXAnswerCallAction after reportNewIncomingCall. Questions Which conditions must hold for iOS to treat the side button as "end call" for a CallKit call, and which is our app failing? Is there a log subsystem or category (callservicesd, TelephonyUtilities) we can read in a sysdiagnose to see why the press is handled as lock? Could WebKit's GPU-process audio session in the same app confuse CallKit about which session carries the call? Would reporting the call at dial time, before any audio, change the outcome? Can ending the in-app call with reportCall(.remoteEnded) instead of a CXEndCallAction leave state in callservicesd that affects the NEXT call? That would fit "works once after a fresh provider, never again". We have no Mac. Is a sysdiagnose captured on the iPhone right after a failing press enough to diagnose this? If you need a project, we can provide the plugin's Swift source (one file) and the Capacitor app project that Codemagic builds.
1
0
87
8h
Public API for torch beam width (Adaptive True Tone flash)?
On iPhone 14 Pro and later, the system flashlight in Control Center can adjust beam width (narrow ↔ wide). As far as I can tell, AVCaptureDevice only exposes torchMode, torchLevel and setTorchModeOn(level:) I found nothing related to beam width Is there any supported way for a third-party app to control the torch beam width? If not, is this on the roadmap?
0
0
144
22h
Improving metric scale consistency of ARWorldTrackingConfiguration on devices without LiDAR
We build a medical scanning app. A practitioner walks a slow orbit (~1 m radius, two laps, 25–35 s) around a standing subject while an ARSession runs, and we capture ~50 high-resolution stills with captureHighResolutionFrame. The ARKit camera poses for those frames are later used to set the metric scale of a photogrammetry reconstruction. We need the resulting scale to be within about 2 % of true. What we observe, measured against a printed A4 sheet on the floor in every scan: Devices with LiDAR (iPhone 16 Pro Max, iPad Pro), using the sceneDepth frame semantic as a check: scale within ±0.8 % across 11 sessions. Devices without LiDAR (iPhone 13, iPad 9th and 11th gen, plus non-LiDAR runs on the Pro devices): the world-frame metre is on average 2–3 % short of true, and varies by a further ±2 % from one session to the next, across 24 sessions in four rooms. Within a session the scale is stable: lap-to-lap variation is under 0.4 %. The error appears to be set during initialisation and held. Nothing we record about capture correlates with the error: time spent before capture, distance walked, vertical camera movement, orbit radius, rotation speed, tracking state, or the residual of our pose fit. Questions: Is there any session setup, initialisation practice, or motion during the first seconds of tracking that is known to improve the metric scale estimate on devices without LiDAR? Does the choice of video format, resolution or frame rate affect scale estimation? Is there any signal, beyond ARCamera.trackingState, that indicates how well scale has been determined for a session, so we could flag sessions rather than accept them? Does anything in the configuration above, in particular locking exposure on the capture device or using the high-resolution video format, work against tracking quality or scale?
1
0
668
23h
Is activeLockedVideoFrameDuration supported on iPhone 16 Pro? Hardware or iOS limitation?
Hello Apple AVFoundation team, Could you clarify the supported iPhone models and OS versions for standalone locked video frame duration using AVCaptureDeviceInput.activeLockedVideoFrameDuration? The API is documented as available from iOS 26.0, but I understand that OS availability alone does not mean every camera device/configuration supports it. I have read the API documentation and the DTS reply in iPhone 17 Pro's Genlock feature, but could not find a model compatibility list for the standalone locked-duration API. Scope: I am investigating actual capture timing for multi-iPhone motion capture. This question is about locking each camera's frame period using its internal timing, without an external genlock accessory. I am not asking about encoding a constant-frame-rate file, ordinary equal min/max frame durations, timecode metadata, or synchronizing the first exposure across phones. Physical-device test iPhone 16 Pro, iOS 26.2 (23C55). Built with Xcode 26.3 and the iOS 26.2 SDK. One AVCaptureSession, one physical rear .builtInWideAngleCamera input, and one AVCaptureVideoDataOutput. 8-bit SDR 420v formats, tested at 1920 x 1080 and 3840 x 2160, each at 30 and 60 fps. Auto video frame rate disabled; no spatial video capture or multicam session. Manual exposure approximately 2 ms, shorter than half a frame at both tested rates. The session is running and delivering frames. Two separate probe runs produced the same results, including capability readbacks after a settling period. For all four tested configurations: input.isLockedVideoFrameDurationSupported = false device.minSupportedLockedVideoFrameDuration = kCMTimeInvalid input.activeLockedVideoFrameDuration = kCMTimeInvalid device.isVideoFrameDurationLocked = falseOrdinary activeVideoMinFrameDuration and activeVideoMaxFrameDuration can both be set to the requested CMTime(value: 1, timescale: 30) or CMTime(value: 1, timescale: 60), and frames are delivered. This is only a configuration readback, not a claim of exact capture timing in the ordinary mode. The app directly queries the public support properties; it has no model whitelist. It does not attempt to set a valid locked duration when support is false, as the documented setter contract would reject that. These tests cover the selected formats, not every camera/format on this phone. Questions Which iPhone models support this standalone API? In particular, is false expected on iPhone 16 Pro? Is there an official compatibility list covering iPhone 13/14/15/16 and iPhone 17 models? Is the iPhone 16 Pro result a hardware capability restriction, an iOS 26.2 limitation/known issue, or a configuration requirement? Is there a released iOS version that enables this on iPhone 16 Pro, or is newer hardware required? What is the minimum supported capture configuration and correct time to query support? Are there restrictions involving the physical versus virtual camera input, pixel format, SDR/HDR, or exposure mode that could explain this result? If so, which configuration should be tested before concluding that the device does not support it? On supported hardware, can this standalone lock be enabled without an external accessory or a special entitlement, independently of follow(_:videoFrameDuration:delegate:)? Relevant documentation: activeLockedVideoFrameDuration isLockedVideoFrameDurationSupported minSupportedLockedVideoFrameDuration I can provide a focused reproducer if needed. Thank you for clarifying the device requirements and whether an OS upgrade can change the result.
1
0
548
3d
iPhone 17 Pro: Is Native Apple Log 2 ISP Processing Available for 4224×3024/60fps RAW?
Hi everyone, I'm developing an iOS camera app that aims to record 4224×3024 Open Gate video at 60fps in 10-bit HEVC with Apple Log 2. Native Apple ISP-processed Apple Log 2 capture works at 3840×2160/60fps. However, at 4224×3024/60fps, the tested capture configuration only provides Bayer RAW (btp2). I've successfully decoded ProRes RAW using Apple's decoder into linear Rec.2020 and applied the published Apple Gamut and Apple Log 2 transforms. However, the resulting chroma is only approximately 72–74% of the native ISP output in controlled color-chart comparisons. I've also confirmed that VTRAWProcessingSession and related RAW processing controls are not publicly available to iOS apps in the iPhoneOS 27.0 SDK. My questions are: Is there a public iOS API that applies the same color and tone rendering as Apple's native camera ISP to captured Bayer RAW or ProRes RAW? Is there a supported configuration for obtaining full-resolution 4224×3024/60fps ISP-processed Apple Log 2 output (x420 or x422), without preview proxies or upscaling? If neither is supported, what is Apple's recommended approach? Is this a hardware limitation or a public API limitation? The goal is to use Apple's official color processing pipeline without empirically fitted matrices, LUTs, or arbitrary color corrections. Any guidance would be greatly appreciated. Thank you!
0
0
265
4d
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
2
0
1.8k
5d
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: 1.On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? 2.Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? 3.For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
2
0
1.6k
5d
TTS Stuttering & Noise
TTS Stuttering & Noise When using Apple’s native TTS API on an iPhone or iPad—specifically while connected to Bluetooth earphones or headsets—users are experiencing audio noise or intermittent gaps in speech, causing parts of sentences to drop out during playback. There were no reports or inquiries regarding such issues up through iOS/iPadOS 26. Since the release of iOS/iPadOS 27, however, there has been a significant increase in reports and inquiries about this problem. If you happen to know the cause of this issue or have any suggestions for a fix, please leave a reply. The issue persists even after removing the AirPods or headset from the device and reconnecting them. The issue persists even after rebooting the device. The issue occurs even on the latest versions, from iOS/iPadOS 27 through 27.2.
0
0
63
5d
AVFoundation: full-resolution Quad Bayer RAW capture path for third-party computational photography? (FB25352517)
At the WWDC26 Camera and Photo Technologies Group Lab (session 8018, ~29:55 to 32:46), the panel explained why ProRAW can produce 48 MP output from Quad Bayer sensors while native Bayer RAW is limited to the binned resolution. ProRAW is already demosaiced and linearized, and may incorporate Photonic Engine multi-frame processing. The native RAW path exposes the binned Bayer result. A true full-resolution RAW output would be Quad Bayer, not conventional Bayer, and supporting it properly needs a decoding path around it. The panel suggested filing Feedback Assistant requests if developers need this, so I filed FB25352517. My use case is computational photography that wants minimally processed sensor-domain samples on purpose — its own multi-frame alignment, denoising, HDR fusion, motion rejection, and super-resolution. Adobe Project Indigo is a public example of that kind of pipeline. It captures RAW bursts and uses measured sub-pixel variation between frames to reconstruct detail. Processed 48 MP photos and ProRAW don't cover that. They've already been through Apple's reconstruction or demosaic. Is filing enhancement requests still the intended path for a supported full-resolution Quad Bayer RAW or mosaic capture path? If so, what would make these reports more useful — mosaic metadata, DNG representation, CIRAWFilter support, AVCaptureDevice.Format capability discovery, something else? WWDC26 session: https://developer.apple.com/videos/play/wwdc2026/8018/ Project Indigo technical overview: https://research.adobe.com/articles/indigo/indigo.html
2
0
350
6d
AVAudioSession ioBufferDuration always default to 512 samples on MacOS27 no matter what is requested
Hello, I develop an audio processing demo app that has been working fine on iOS/iPad and on Mac ("developed for iPad") until I upgraded to MacOS27. The app is demonstrating low latency / high temporal resolution audio spectral analysis. I am using AVAudioSession, I request the audio IO buffer to be 32 samples. This still works fine on iOS/iPadOS, but it seems that when running on MacOS27 the buffer size always defaults to 512 samples no matter what I request (smaller or larger). This applies to the "Designed for iPad" version running on MacOS and any version running in the iPhone and iPad simulators on MacOS. The app seems to be working as intended on physical iPhones and iPads. I made a catalyst version and used AudioObjectSetPropertyData to force the IO bugger sizes to 32 samples on the system's active hardware device. This seems to succeed but still results in incorrect behaviour in my app; I am still diagnosing but I believe it's because this leaves AVAudioSession with incorrect data, and I am using AudioKit which relies on AVAudioSession (not confirmed). Regardless, is this change in behaviour intentional in MacOS27? Is there a way around it? The whole point of my app is to demonstrate efficient audio processing with very low latency - I need small input buffer size for that. Thank you for any help with this
1
0
305
1w
Official API for Apple Log / Apple Log 2 to SDR Rec.709 and HDR HLG Conversion?
Hello, I'm developing a video application for Apple platforms and would like to implement accurate Apple Log and Apple Log 2 color conversion using official Apple APIs. Final Cut Pro and Final Cut Camera support SDR Rec.709 and HDR Rec.2020 HLG monitoring. I have found Apple's official Apple Log white papers and Rec.709 LUTs, but I could not find an official HDR HLG LUT or a public API that provides equivalent color conversion. I have three questions: Does Apple provide a public API through AVFoundation, Core Image, ColorSync, or another framework to convert Apple Log / Apple Log 2 to SDR Rec.709 and HDR Rec.2020 HLG? Are the color transformations used by Final Cut Pro and Final Cut Camera available to third-party developers, either through public APIs or official LUT resources? If no such API exists, is implementing the transfer functions and gamut conversions from Apple's white papers, followed by standard SDR/HDR rendering, the recommended approach? My goal is to preserve the original Apple Log footage while allowing independent SDR/HDR monitoring without modifying the recorded data. Any guidance, documentation, or sample code from Apple engineers would be greatly appreciated. Thank you!
1
0
281
1w
AVAudioSession blocks 20–36s on A2DP→HFP switch while another AudioUnit is rendering (iOS 27.0.1, AirPods Spatial Audio)
While an AVAudioPlayer/AudioUnit is rendering to AirPods over A2DP with system Spatial Audio + Head Tracking enabled, switching the shared AVAudioSession to playAndRecord + voiceChat (A2DP → HFP) blocks the main thread for tens of seconds on synchronous AVAudioSession APIs (setCategory / setMode / channel getters / setPreferred*). iOS hang detection reports a Severe Hang of 41.02 s. Stopping playback and deactivating the session before the identical switch completes in ~87 ms. Feedback: FB25324099 Environment iPhone 15 Pro (iPhone16,1), iOS 27.0.1 (24A446) AirPods worn; Spatial Audio + Head Tracking on AVFoundation-only repro (no third-party SDK): https://github.com/shinobuwz/ios27-spatial-audio-stall-repro Info.plist: LSApplicationCategoryType = public.app-category.games, LSSupportsGameMode = YES; does NOT set AVGameBypassSystemSpatialAudio (those two keys are required for system Spatial Audio on this path) Steps Wear AirPods with Spatial Audio + Head Tracking enabled Launch the repro app; allow mic (or use auto-repro) Play 440 Hz (BluetoothA2DP, spatial=1) After ~2 s, run Repro: keep playing and switch to PlayAndRecord + VoiceChat Compare with Control: stop playback + setActive:NO, then the same switch Expected: each call returns in ms; whole reconfiguration ≪ 50 ms Actual (cold launches): configuration total 20.8–28.9 s (options=13); with options=237 ≈ 36.2 s; first input buffer ~40–58 s from switch start. Control ≈ 87 ms. Evidence in the repo: docs/MEASUREMENTS-2026-10-26.md, evidence/*.log, hangs.xml, main-thread stacks in avas::client::SessionCore (sync XPC to the audio daemon). Even inputNumberOfChannels / outputNumberOfChannels getters stall for seconds. Question: Is this a known regression on iOS 27? Any workaround besides stopping all rendering AudioUnits before the HFP switch? Happy to attach sysdiagnose / Instruments / screen recording if useful.
0
0
423
1w
iOS 27 beta: Opening Notification Center pauses AVPlayer playback
Since iOS 27 beta we are seeing a behavior change in our video streaming app and I would like to know whether others can reproduce it. Behavior on iOS 27 beta: Fully opening the Notification Center pauses playback. Audio continues for about 5 more seconds, then the app is suspended. Closing the Notification Center leaves the player paused. On iOS 26 the same build keeps playing in this situation. Setup: AVQueuePlayer playing video with audio, not muted audiovisualBackgroundPlaybackPolicy = .automatic (default) AVAudioSession category .playback, UIBackgroundModes: audio entitlement AVPictureInPictureController attached with canStartPictureInPictureAutomaticallyFromInline = true Filed as FB23893965 Questions: Can anyone reproduce this on iOS 27 beta? Is this intentional or a regression?
3
5
1.3k
1w
AVAudioRecorder loses earlier audio after Siri with iOS 27 lifecycle notifications
I’m migrating to the new notifications described in Handling audio interruptions. On an iPhone 14 running iOS 27.0.1 (24A446), I call pause() when the system deactivates the session. On .shouldResume, I await successful session.activate() and call record() on the same recorder. It returns true, but the saved file contains only the audio recorded after Siri. To reproduce: record for five seconds, invoke and dismiss Siri, record for another five seconds, then stop and play. The earlier five seconds are missing. A legacy-notification control preserved both portions on the same phone. Is another step required to preserve the recording with the new notifications, or is this a known issue? I found this similar report, although it uses the legacy API. Minimal reproduction Built with Xcode 27.1. Create a SwiftUI iOS app targeting iOS 27, replace its generated App.swift with the code below, and add these Info.plist entries: NSMicrophoneUsageDescription: Test audio recording UIBackgroundModes: an array containing audio import SwiftUI import AVFoundation @main struct RecorderReproApp: App { var body: some Scene { WindowGroup { RecorderReproView() } } } @MainActor struct RecorderReproView: View { private let session = AVAudioSession.sharedInstance() private let file = URL.documentsDirectory.appendingPathComponent("recording.m4a") @State private var recorder: AVAudioRecorder? @State private var player: AVAudioPlayer? @State private var running = false @State private var interrupted = false @State private var status = "Ready" var body: some View { VStack(spacing: 24) { Button("Start") { Task { await start() } }.disabled(running) Button("Stop") { running = false interrupted = false recorder?.stop() status = file.path }.disabled(!running) Button("Play") { do { player = try AVAudioPlayer(contentsOf: file) status = "Saved duration: \(player!.duration) seconds" player?.play() } catch { status = error.localizedDescription } }.disabled(running) Text(status) } .padding() .task { for await notification in NotificationCenter.default.notifications( named: AVAudioSession.didBecomeInactiveNotification, object: session ) { guard running, let context = notification.userInfo?[AVAudioSession.deactivationContextKey] as? AVAudioSession.DeactivationContext, context.source == .system else { continue } recorder?.pause() interrupted = true status = "Interrupted" } } .task { for await notification in NotificationCenter.default.notifications( named: AVAudioSession.resumptionRecommendationNotification, object: session ) { guard running, interrupted, let context = notification.userInfo?[AVAudioSession.resumptionContextKey] as? AVAudioSession.ResumptionContext else { continue } guard context.recommendation == .shouldResume else { interrupted = false continue } do { guard try await session.activate(), running, interrupted else { continue } status = "Resumed: \(recorder!.record())" interrupted = false } catch { status = error.localizedDescription } } } } private func start() async { guard await AVAudioApplication.requestRecordPermission() else { return } do { try session.setCategory(.playAndRecord) guard try await session.activate() else { return } recorder = try AVAudioRecorder(url: file, settings: [ AVFormatIDKey: kAudioFormatMPEG4AAC_HE, AVSampleRateKey: 44_100, AVEncoderBitRateKey: 32_000, AVNumberOfChannelsKey: 2 ]) running = recorder!.record() status = "Recording: \(running)" } catch { status = error.localizedDescription } } }
0
0
451
1w
iPhone Duo: Does CameraCaptureAccessory accept touch, what counts as an active camera session, and can both front cameras run at once?
After watching "Leverage multiple displays and scenes on iPhone Duo" and "Build a great camera experience for iPhone Duo", we have four questions about the outer display while the device is open: Does a view hosted in CameraCaptureAccessory receive touch input on the outer display, or is it display-only? The accessory is described as available when the app "has an active camera session." Does that require a running preview layer or an attached output (photo, movie or data), or is a running AVCaptureSession with an input enough? Is a session whose main purpose is to host the accessory a supported use? Can AVCaptureMultiCamSession run the inner (under-display) front camera and the outer front camera at the same time on iPhone Duo? If so, which device-type combinations are supported, and are there thermal or power limits? Is CameraCaptureAccessory the only way for an app to show content on the outer display while the inner display is live, or are scene accessories that don't depend on the camera planned? Our use case is not photography. We're exploring a companion view for a second person sitting across from the device owner, so we want to understand whether the camera accessory is the right tool before designing around it.
5
1
352
2w
Why does Apple prefer Quad Bayer over traditional Bayer sensors?
Hi everyone, I've been reading about iPhone camera sensors recently, and there's something I've been wondering about. Apple has explained the benefits of Quad Bayer, especially pixel binning for low-light photography and high-resolution capture. But why not use a traditional Bayer sensor with Dual Pixel PDAF instead? As far as I understand, a native Bayer arrangement could potentially preserve more fine detail without requiring remosaicing. And if both sensors have the same size and pixel count, couldn't the Bayer sensor also produce a cleaner 12MP image by downsampling? I'm also curious about the microlens design. Some sensors use one microlens shared by four pixels for phase detection. Wouldn't individual microlenses potentially offer better spatial resolution? So my main question is: What hardware-level advantages make Quad Bayer worth choosing over traditional Bayer, despite the additional image reconstruction involved? Is it mainly about read noise, analog binning, power efficiency, autofocus, or something else? I'm not a camera engineer, just someone interested in understanding the technology behind iPhone cameras. I'd really appreciate any technical insight. Thanks!
0
0
108
2w
Feature request: level metering for MusicKit playback (FB25234486)
Hi everyone, I'm building a tvOS app that shows analog VU meters for the music playing, in the style of vintage hi-fi amplifiers and tape decks. It works well with local and network files played through AVAudioEngine, where I can tap the signal and measure real levels. With Apple Music, though, there's no way to get even basic level information. The content is DRM-protected, SystemMusicPlayer and ApplicationMusicPlayer render audio outside the app's audio graph, and MusicKit has no metering API. My current workaround is to analyze the song's 30-second preview and replay its level envelope in a loop. It looks plausible, but it isn't in sync with the actual music. I filed FB25234486 asking for a read-only metering API on MusicPlayer, similar to what AVAudioPlayer already offers: let rms = musicPlayer.averagePower(forChannel: 0) // dBFS let peak = musicPlayer.peakPower(forChannel: 0) // dBFS This wouldn't expose any audio samples: just one RMS and one peak value per channel, at a low, capped rate (around 30 Hz). Those numbers can't be turned back into audio, so content protection wouldn't be weakened, and the values could be computed inside the system's playback process. It would make a number of use cases possible: VU and peak meters, audio-reactive visualizers, ambient lighting synced to the music (HomeKit, smart lights), loudness displays, and accessibility features such as visual feedback of the music for hard-of-hearing users. If you need this too, please file your own feedback and mention FB25234486. Duplicate reports help Apple gauge demand. And if anyone has found a supported way to get playback levels from MusicKit today, I'd love to hear about it. Thanks!
0
0
127
2w
Ultra Wide camera: frame-to-frame jitter below 2.0x and sharpness step at 2.0x during native zoom ramp (iPhone 17 Pro, 4K/60)
I am developing CineZoom, part of CameraControl, a remote camera-direction system (macOS app controlling an iPhone capture engine). Its core requirement is a smooth, repeatable cinematic zoom. On iPhone 17 Pro (iOS 26.5.1), using the physical Back Ultra Wide camera at 3840x2160, 60 fps, a single native ramp(toVideoZoomFactor:withRate:) from 1.7x to 2.1x over 5 seconds shows a visible stutter. I isolated it with a minimal diagnostic app (no networking) and analysed four tripod recordings frame by frame, in two locations with different lighting. The results are consistent across all four: Jitter below 2.0x: while the zoom factor is below 2.0, the image centre shifts about 2.8 px RMS between consecutive frames (peaks ~5 px), alternating almost every frame (26-28 Hz). Above 2.0x, during the same ramp at the same speed, it drops to 1.2-1.5 px. Sharpness step at 2.0x: centre-crop sharpness drops by 23-31% within a few frames exactly when the zoom factor crosses 2.0. There is no consistent effect at the reported secondaryNativeResolutionZoomFactors value (1.7838). The zoom command path itself is correct: the zoom measured from the image matches videoZoomFactor throughout, with no dropped or duplicated frames. Questions: Is this expected behaviour for the Ultra Wide camera in this zoom range? Is the jitter related to video stabilization or to how the sensor crop is positioned in this readout mode? Is there a public API that reports or controls the zoom factor at which this transition happens, so an app can plan around it? What configuration do you recommend for a smooth continuous video zoom across 1.7x-2.1x on this camera (format, stabilization mode, constituent-camera switching)? Are there existing APIs that would let an app detect or compensate this (per-frame stabilization/crop metadata, consistent processing during zoom, or a reduced-processing video stream)? I have a full technical report, per-frame measurements, charts and the original recordings, and I am happy to share them. Thank you!
1
0
694
2w
AirPods camera-remote events for an external-camera controller
We are developing CameraDirector, an iPhone application that controls external professional cameras through documented vendor APIs and reads back native recording state. We want an AirPods camera-remote press to request record start/stop on the selected external camera, then provide audible confirmation only after the camera reports its state. In external-camera control mode, the app would be in the foreground but would not be capturing video from an iPhone camera. The WWDC25 session 'Enhancing your camera experience with capture controls' states that AVCaptureEventInteraction (or SwiftUI's onCameraCaptureEvent) requires an active AVCaptureSession and does not receive events when the app is backgrounded: https://developer.apple.com/videos/play/wwdc2025/253/ Is there a supported configuration for receiving AirPods capture events in this external-camera mode without running a local iPhone camera capture? Does an active audio-only AVCaptureSession qualify, or does event delivery require an active video-camera input? If this is outside the intended API contract, is there another supported input path for this use case, or should we file an enhancement request? We have not yet established an on-device AirPods result for this configuration. We would appreciate guidance on the intended architecture or a relevant sample and can provide a focused reproducer for an implementation issue. Related capture-event implementation discussion: https://developer.apple.com/forums/thread/740616 Thank you, Eddie DaRoza ACTION Cinema Technologies
1
0
735
2w
AVSpeechSynthesizer pulls words out of thin air.
Hi, I'm working on a project that uses the AVSpeechSynthesizer and AVSpeechUtterance. I discovered by chance that the AVSpeechSynthesizer automatically completes some words instead of just outputting what it's supposed to. These are abbreviations for days of the week or months, but not all of them. I don't want either of them automatically completed, but only the specified text. The completion transcends languages. I have written a short example program for demonstration purposes. import SwiftUI import AVFoundation import Foundation let synthesizer: AVSpeechSynthesizer = AVSpeechSynthesizer() struct ContentView: View { var body: some View { VStack { Button { utter("mon") } label: { Text("mon") } .buttonStyle(.borderedProminent) Button { utter("tue") } label: { Text("tue") } .buttonStyle(.borderedProminent) Button { utter("thu") } label: { Text("thu") } .buttonStyle(.borderedProminent) Button { utter("feb") } label: { Text("feb") } .buttonStyle(.borderedProminent) Button { utter("feb", lang: "de-DE") } label: { Text("feb DE") } .buttonStyle(.borderedProminent) Button { utter("wed") } label: { Text("wed") } .buttonStyle(.borderedProminent) } .padding() } private func utter(_ text: String, lang: String = "en-US") { let utterance = AVSpeechUtterance(string: text) let voice = AVSpeechSynthesisVoice(language: lang) utterance.voice = voice synthesizer.speak(utterance) } } #Preview { ContentView() } Thank you Christian
4
0
1.6k
3w
CallKit VoIP call on the receiver: the side (lock) button only locks the screen, no CXEndCallAction is delivered
Device and versions iPhone 15, iOS 27.0.1 App: Hush, a private messenger. Capacitor 8.5.2 iOS app; the UI is a web page in a WKWebView. Call signalling runs in the page; call audio runs in a native Swift Capacitor plugin on stasel/WebRTC 154.0.0, integrated with CallKit. Built on Codemagic with the latest Xcode, distributed through TestFlight. UIBackgroundModes: audio, voip. No PushKit yet: calls are placed and answered in the app's own screen while it is in the foreground. Setup RTCAudioSession manual audio (useManualAudio = true). isAudioEnabled is set only after ICE connects AND provider(_:didActivate:) is called (a 3-second fallback exists but never fired in these tests). We call RTCAudioSession.audioSessionDidActivate / audioSessionDidDeactivate from the provider delegate. Session: playAndRecord, mode voiceChat, options allowBluetoothHFP, no defaultToSpeaker. overrideOutputAudioPort is only called when the user picks Speaker (never in the failing tests). One CXProvider, created lazily, delegate on the main queue. Configuration: supportsVideo false (also tested true), maximumCallGroups 1, maximumCallsPerCallGroup 1, supportedHandleTypes [.generic], includesCallsInRecents false (also tested true). Outgoing: CXStartCallAction is requested via CXCallController when ICE first connects (not at dial time; the page plays its own ringback until then), handle generic, value "Hush". In perform(CXStartCallAction) we configure the audio session (no activation), reportCall(with:updated:), fulfil, then reportOutgoingCall(startedConnectingAt:) and (connectedAt:). Answering side: reportNewIncomingCall, then an immediate CXAnswerCallAction (the user already accepted in the app's screen). Mute: CXSetMutedCallAction. In-app End reported the call with reportCall(with:endedAt:reason: .remoteEnded) in all tests below (we are changing this to a CXEndCallAction request; not yet tested). Proximity: isProximityMonitoringEnabled = true from dialling while on the receiver. The screen does go dark at the ear. Also tested with our proximity monitoring off. The problem With a Hush call live, route Receiver, CallKit holding the call (the system call UI shows it), didActivate delivered exactly once, our audio running inside that activation, proximity on and the screen dark at the ear: pressing the side button only locks the screen. No CXEndCallAction arrives and the call continues. In the Phone app on the same iPhone the side button ends the call at the ear. Snapchat calls on this iPhone end on the side button while Snapchat's call screen is open. What works: End on the system lock-screen call UI delivers a CXEndCallAction and we end the call. Mute from the system UI arrives as CXSetMutedCallAction. So provider delegate delivery works. What we tried, in order (each tested on device, with a readout of the plugin's state after the call) First CallKit build: side button did nothing; lock-screen End worked. Readout: "CallKit off, its audio off (0x), fallback no, route speaker, proximity on". Cause: CXStartCallAction was refused (missing voip background mode) and the error swallowed. Fixed: voip added; on CXErrorCodeRequestTransactionError.unknownCallProvider we invalidate and recreate the provider and retry once. Next build: CallKit held the call, didActivate once, route Receiver. Side button still did nothing, no CXEndCallAction. Our route log showed two route changes caused by our own overrideOutputAudioPort(.none) on every activation and route change. Removed: no override unless Speaker is selected. Result: no change. Diagnostic switches, tested in combinations: our proximity monitoring on/off; supportsVideo false/true; includesCallsInRecents false/true; callee now reported with reportNewIncomingCall + CXAnswerCallAction (previously both sides used CXStartCallAction). Result: the side button ended the call EXACTLY ONCE: the first call after the provider was recreated (changing supportsVideo makes us invalidate() and create a new CXProvider). Every later call with the same configuration failed. Readout for the failures: "CallKit on, its audio on (1x), fallback no, route Receiver, proximity on", screen dark at the ear. Following thread 837211 (creating a provider implicitly pushes the audio session ID to callservicesd, which can keep a stale one; set the configuration before reporting each call): we now assign provider.configuration = a fresh CXProviderConfiguration at call start and again immediately before each CXStartCallAction request / reportNewIncomingCall. We also added a switch that recreates the CXProvider for every call. With the re-push alone: no end action. With a fresh provider every call: no end action either. Readout each time: "provider calls 1, config re-pushed yes, CallKit on, its audio on (1x), route Receiver, proximity on". The single success in step 3 has not been reproduced. Not tried on purpose: detecting the lock ourselves (protectedDataWillBecomeUnavailable arrives about 10 seconds late and needs a passcode; a 2017 thread says Home and Lock cannot be told apart); reporting the CallKit call at dial time instead of ICE connect; PushKit. Threads we relied on 819363 (April 2026): the lock button hangs up a CallKit call handled directly by the device (not through a headset), delivered as the provider's end action. 837211 (August 2026): stale audio session ID in callservicesd after provider creation; set the configuration before reporting a call. How we differ from apps where it works WKWebView: WebKit's GPU process holds a second AVAudioSession in our app's name. During the call the page captures no mic (the plugin owns it); before connect the page plays a Web Audio ringback (Ambient category). The CallKit call starts at ICE connect, not at dial time. In-app End reported as .remoteEnded rather than requested as a CXEndCallAction. The callee's immediate CXAnswerCallAction after reportNewIncomingCall. Questions Which conditions must hold for iOS to treat the side button as "end call" for a CallKit call, and which is our app failing? Is there a log subsystem or category (callservicesd, TelephonyUtilities) we can read in a sysdiagnose to see why the press is handled as lock? Could WebKit's GPU-process audio session in the same app confuse CallKit about which session carries the call? Would reporting the call at dial time, before any audio, change the outcome? Can ending the in-app call with reportCall(.remoteEnded) instead of a CXEndCallAction leave state in callservicesd that affects the NEXT call? That would fit "works once after a fresh provider, never again". We have no Mac. Is a sysdiagnose captured on the iPhone right after a failing press enough to diagnose this? If you need a project, we can provide the plugin's Swift source (one file) and the Capacitor app project that Codemagic builds.
Replies
1
Boosts
0
Views
87
Activity
8h
Public API for torch beam width (Adaptive True Tone flash)?
On iPhone 14 Pro and later, the system flashlight in Control Center can adjust beam width (narrow ↔ wide). As far as I can tell, AVCaptureDevice only exposes torchMode, torchLevel and setTorchModeOn(level:) I found nothing related to beam width Is there any supported way for a third-party app to control the torch beam width? If not, is this on the roadmap?
Replies
0
Boosts
0
Views
144
Activity
22h
Improving metric scale consistency of ARWorldTrackingConfiguration on devices without LiDAR
We build a medical scanning app. A practitioner walks a slow orbit (~1 m radius, two laps, 25–35 s) around a standing subject while an ARSession runs, and we capture ~50 high-resolution stills with captureHighResolutionFrame. The ARKit camera poses for those frames are later used to set the metric scale of a photogrammetry reconstruction. We need the resulting scale to be within about 2 % of true. What we observe, measured against a printed A4 sheet on the floor in every scan: Devices with LiDAR (iPhone 16 Pro Max, iPad Pro), using the sceneDepth frame semantic as a check: scale within ±0.8 % across 11 sessions. Devices without LiDAR (iPhone 13, iPad 9th and 11th gen, plus non-LiDAR runs on the Pro devices): the world-frame metre is on average 2–3 % short of true, and varies by a further ±2 % from one session to the next, across 24 sessions in four rooms. Within a session the scale is stable: lap-to-lap variation is under 0.4 %. The error appears to be set during initialisation and held. Nothing we record about capture correlates with the error: time spent before capture, distance walked, vertical camera movement, orbit radius, rotation speed, tracking state, or the residual of our pose fit. Questions: Is there any session setup, initialisation practice, or motion during the first seconds of tracking that is known to improve the metric scale estimate on devices without LiDAR? Does the choice of video format, resolution or frame rate affect scale estimation? Is there any signal, beyond ARCamera.trackingState, that indicates how well scale has been determined for a session, so we could flag sessions rather than accept them? Does anything in the configuration above, in particular locking exposure on the capture device or using the high-resolution video format, work against tracking quality or scale?
Replies
1
Boosts
0
Views
668
Activity
23h
Is activeLockedVideoFrameDuration supported on iPhone 16 Pro? Hardware or iOS limitation?
Hello Apple AVFoundation team, Could you clarify the supported iPhone models and OS versions for standalone locked video frame duration using AVCaptureDeviceInput.activeLockedVideoFrameDuration? The API is documented as available from iOS 26.0, but I understand that OS availability alone does not mean every camera device/configuration supports it. I have read the API documentation and the DTS reply in iPhone 17 Pro's Genlock feature, but could not find a model compatibility list for the standalone locked-duration API. Scope: I am investigating actual capture timing for multi-iPhone motion capture. This question is about locking each camera's frame period using its internal timing, without an external genlock accessory. I am not asking about encoding a constant-frame-rate file, ordinary equal min/max frame durations, timecode metadata, or synchronizing the first exposure across phones. Physical-device test iPhone 16 Pro, iOS 26.2 (23C55). Built with Xcode 26.3 and the iOS 26.2 SDK. One AVCaptureSession, one physical rear .builtInWideAngleCamera input, and one AVCaptureVideoDataOutput. 8-bit SDR 420v formats, tested at 1920 x 1080 and 3840 x 2160, each at 30 and 60 fps. Auto video frame rate disabled; no spatial video capture or multicam session. Manual exposure approximately 2 ms, shorter than half a frame at both tested rates. The session is running and delivering frames. Two separate probe runs produced the same results, including capability readbacks after a settling period. For all four tested configurations: input.isLockedVideoFrameDurationSupported = false device.minSupportedLockedVideoFrameDuration = kCMTimeInvalid input.activeLockedVideoFrameDuration = kCMTimeInvalid device.isVideoFrameDurationLocked = falseOrdinary activeVideoMinFrameDuration and activeVideoMaxFrameDuration can both be set to the requested CMTime(value: 1, timescale: 30) or CMTime(value: 1, timescale: 60), and frames are delivered. This is only a configuration readback, not a claim of exact capture timing in the ordinary mode. The app directly queries the public support properties; it has no model whitelist. It does not attempt to set a valid locked duration when support is false, as the documented setter contract would reject that. These tests cover the selected formats, not every camera/format on this phone. Questions Which iPhone models support this standalone API? In particular, is false expected on iPhone 16 Pro? Is there an official compatibility list covering iPhone 13/14/15/16 and iPhone 17 models? Is the iPhone 16 Pro result a hardware capability restriction, an iOS 26.2 limitation/known issue, or a configuration requirement? Is there a released iOS version that enables this on iPhone 16 Pro, or is newer hardware required? What is the minimum supported capture configuration and correct time to query support? Are there restrictions involving the physical versus virtual camera input, pixel format, SDR/HDR, or exposure mode that could explain this result? If so, which configuration should be tested before concluding that the device does not support it? On supported hardware, can this standalone lock be enabled without an external accessory or a special entitlement, independently of follow(_:videoFrameDuration:delegate:)? Relevant documentation: activeLockedVideoFrameDuration isLockedVideoFrameDurationSupported minSupportedLockedVideoFrameDuration I can provide a focused reproducer if needed. Thank you for clarifying the device requirements and whether an OS upgrade can change the result.
Replies
1
Boosts
0
Views
548
Activity
3d
iPhone 17 Pro: Is Native Apple Log 2 ISP Processing Available for 4224×3024/60fps RAW?
Hi everyone, I'm developing an iOS camera app that aims to record 4224×3024 Open Gate video at 60fps in 10-bit HEVC with Apple Log 2. Native Apple ISP-processed Apple Log 2 capture works at 3840×2160/60fps. However, at 4224×3024/60fps, the tested capture configuration only provides Bayer RAW (btp2). I've successfully decoded ProRes RAW using Apple's decoder into linear Rec.2020 and applied the published Apple Gamut and Apple Log 2 transforms. However, the resulting chroma is only approximately 72–74% of the native ISP output in controlled color-chart comparisons. I've also confirmed that VTRAWProcessingSession and related RAW processing controls are not publicly available to iOS apps in the iPhoneOS 27.0 SDK. My questions are: Is there a public iOS API that applies the same color and tone rendering as Apple's native camera ISP to captured Bayer RAW or ProRes RAW? Is there a supported configuration for obtaining full-resolution 4224×3024/60fps ISP-processed Apple Log 2 output (x420 or x422), without preview proxies or upscaling? If neither is supported, what is Apple's recommended approach? Is this a hardware limitation or a public API limitation? The goal is to use Apple's official color processing pipeline without empirically fitted matrices, LUTs, or arbitrary color corrections. Any guidance would be greatly appreciated. Thank you!
Replies
0
Boosts
0
Views
265
Activity
4d
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
Replies
2
Boosts
0
Views
1.8k
Activity
5d
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: 1.On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? 2.Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? 3.For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
Replies
2
Boosts
0
Views
1.6k
Activity
5d
TTS Stuttering & Noise
TTS Stuttering & Noise When using Apple’s native TTS API on an iPhone or iPad—specifically while connected to Bluetooth earphones or headsets—users are experiencing audio noise or intermittent gaps in speech, causing parts of sentences to drop out during playback. There were no reports or inquiries regarding such issues up through iOS/iPadOS 26. Since the release of iOS/iPadOS 27, however, there has been a significant increase in reports and inquiries about this problem. If you happen to know the cause of this issue or have any suggestions for a fix, please leave a reply. The issue persists even after removing the AirPods or headset from the device and reconnecting them. The issue persists even after rebooting the device. The issue occurs even on the latest versions, from iOS/iPadOS 27 through 27.2.
Replies
0
Boosts
0
Views
63
Activity
5d
AVFoundation: full-resolution Quad Bayer RAW capture path for third-party computational photography? (FB25352517)
At the WWDC26 Camera and Photo Technologies Group Lab (session 8018, ~29:55 to 32:46), the panel explained why ProRAW can produce 48 MP output from Quad Bayer sensors while native Bayer RAW is limited to the binned resolution. ProRAW is already demosaiced and linearized, and may incorporate Photonic Engine multi-frame processing. The native RAW path exposes the binned Bayer result. A true full-resolution RAW output would be Quad Bayer, not conventional Bayer, and supporting it properly needs a decoding path around it. The panel suggested filing Feedback Assistant requests if developers need this, so I filed FB25352517. My use case is computational photography that wants minimally processed sensor-domain samples on purpose — its own multi-frame alignment, denoising, HDR fusion, motion rejection, and super-resolution. Adobe Project Indigo is a public example of that kind of pipeline. It captures RAW bursts and uses measured sub-pixel variation between frames to reconstruct detail. Processed 48 MP photos and ProRAW don't cover that. They've already been through Apple's reconstruction or demosaic. Is filing enhancement requests still the intended path for a supported full-resolution Quad Bayer RAW or mosaic capture path? If so, what would make these reports more useful — mosaic metadata, DNG representation, CIRAWFilter support, AVCaptureDevice.Format capability discovery, something else? WWDC26 session: https://developer.apple.com/videos/play/wwdc2026/8018/ Project Indigo technical overview: https://research.adobe.com/articles/indigo/indigo.html
Replies
2
Boosts
0
Views
350
Activity
6d
AVAudioSession ioBufferDuration always default to 512 samples on MacOS27 no matter what is requested
Hello, I develop an audio processing demo app that has been working fine on iOS/iPad and on Mac ("developed for iPad") until I upgraded to MacOS27. The app is demonstrating low latency / high temporal resolution audio spectral analysis. I am using AVAudioSession, I request the audio IO buffer to be 32 samples. This still works fine on iOS/iPadOS, but it seems that when running on MacOS27 the buffer size always defaults to 512 samples no matter what I request (smaller or larger). This applies to the "Designed for iPad" version running on MacOS and any version running in the iPhone and iPad simulators on MacOS. The app seems to be working as intended on physical iPhones and iPads. I made a catalyst version and used AudioObjectSetPropertyData to force the IO bugger sizes to 32 samples on the system's active hardware device. This seems to succeed but still results in incorrect behaviour in my app; I am still diagnosing but I believe it's because this leaves AVAudioSession with incorrect data, and I am using AudioKit which relies on AVAudioSession (not confirmed). Regardless, is this change in behaviour intentional in MacOS27? Is there a way around it? The whole point of my app is to demonstrate efficient audio processing with very low latency - I need small input buffer size for that. Thank you for any help with this
Replies
1
Boosts
0
Views
305
Activity
1w
Official API for Apple Log / Apple Log 2 to SDR Rec.709 and HDR HLG Conversion?
Hello, I'm developing a video application for Apple platforms and would like to implement accurate Apple Log and Apple Log 2 color conversion using official Apple APIs. Final Cut Pro and Final Cut Camera support SDR Rec.709 and HDR Rec.2020 HLG monitoring. I have found Apple's official Apple Log white papers and Rec.709 LUTs, but I could not find an official HDR HLG LUT or a public API that provides equivalent color conversion. I have three questions: Does Apple provide a public API through AVFoundation, Core Image, ColorSync, or another framework to convert Apple Log / Apple Log 2 to SDR Rec.709 and HDR Rec.2020 HLG? Are the color transformations used by Final Cut Pro and Final Cut Camera available to third-party developers, either through public APIs or official LUT resources? If no such API exists, is implementing the transfer functions and gamut conversions from Apple's white papers, followed by standard SDR/HDR rendering, the recommended approach? My goal is to preserve the original Apple Log footage while allowing independent SDR/HDR monitoring without modifying the recorded data. Any guidance, documentation, or sample code from Apple engineers would be greatly appreciated. Thank you!
Replies
1
Boosts
0
Views
281
Activity
1w
AVAudioSession blocks 20–36s on A2DP→HFP switch while another AudioUnit is rendering (iOS 27.0.1, AirPods Spatial Audio)
While an AVAudioPlayer/AudioUnit is rendering to AirPods over A2DP with system Spatial Audio + Head Tracking enabled, switching the shared AVAudioSession to playAndRecord + voiceChat (A2DP → HFP) blocks the main thread for tens of seconds on synchronous AVAudioSession APIs (setCategory / setMode / channel getters / setPreferred*). iOS hang detection reports a Severe Hang of 41.02 s. Stopping playback and deactivating the session before the identical switch completes in ~87 ms. Feedback: FB25324099 Environment iPhone 15 Pro (iPhone16,1), iOS 27.0.1 (24A446) AirPods worn; Spatial Audio + Head Tracking on AVFoundation-only repro (no third-party SDK): https://github.com/shinobuwz/ios27-spatial-audio-stall-repro Info.plist: LSApplicationCategoryType = public.app-category.games, LSSupportsGameMode = YES; does NOT set AVGameBypassSystemSpatialAudio (those two keys are required for system Spatial Audio on this path) Steps Wear AirPods with Spatial Audio + Head Tracking enabled Launch the repro app; allow mic (or use auto-repro) Play 440 Hz (BluetoothA2DP, spatial=1) After ~2 s, run Repro: keep playing and switch to PlayAndRecord + VoiceChat Compare with Control: stop playback + setActive:NO, then the same switch Expected: each call returns in ms; whole reconfiguration ≪ 50 ms Actual (cold launches): configuration total 20.8–28.9 s (options=13); with options=237 ≈ 36.2 s; first input buffer ~40–58 s from switch start. Control ≈ 87 ms. Evidence in the repo: docs/MEASUREMENTS-2026-10-26.md, evidence/*.log, hangs.xml, main-thread stacks in avas::client::SessionCore (sync XPC to the audio daemon). Even inputNumberOfChannels / outputNumberOfChannels getters stall for seconds. Question: Is this a known regression on iOS 27? Any workaround besides stopping all rendering AudioUnits before the HFP switch? Happy to attach sysdiagnose / Instruments / screen recording if useful.
Replies
0
Boosts
0
Views
423
Activity
1w
iOS 27 beta: Opening Notification Center pauses AVPlayer playback
Since iOS 27 beta we are seeing a behavior change in our video streaming app and I would like to know whether others can reproduce it. Behavior on iOS 27 beta: Fully opening the Notification Center pauses playback. Audio continues for about 5 more seconds, then the app is suspended. Closing the Notification Center leaves the player paused. On iOS 26 the same build keeps playing in this situation. Setup: AVQueuePlayer playing video with audio, not muted audiovisualBackgroundPlaybackPolicy = .automatic (default) AVAudioSession category .playback, UIBackgroundModes: audio entitlement AVPictureInPictureController attached with canStartPictureInPictureAutomaticallyFromInline = true Filed as FB23893965 Questions: Can anyone reproduce this on iOS 27 beta? Is this intentional or a regression?
Replies
3
Boosts
5
Views
1.3k
Activity
1w
AVAudioRecorder loses earlier audio after Siri with iOS 27 lifecycle notifications
I’m migrating to the new notifications described in Handling audio interruptions. On an iPhone 14 running iOS 27.0.1 (24A446), I call pause() when the system deactivates the session. On .shouldResume, I await successful session.activate() and call record() on the same recorder. It returns true, but the saved file contains only the audio recorded after Siri. To reproduce: record for five seconds, invoke and dismiss Siri, record for another five seconds, then stop and play. The earlier five seconds are missing. A legacy-notification control preserved both portions on the same phone. Is another step required to preserve the recording with the new notifications, or is this a known issue? I found this similar report, although it uses the legacy API. Minimal reproduction Built with Xcode 27.1. Create a SwiftUI iOS app targeting iOS 27, replace its generated App.swift with the code below, and add these Info.plist entries: NSMicrophoneUsageDescription: Test audio recording UIBackgroundModes: an array containing audio import SwiftUI import AVFoundation @main struct RecorderReproApp: App { var body: some Scene { WindowGroup { RecorderReproView() } } } @MainActor struct RecorderReproView: View { private let session = AVAudioSession.sharedInstance() private let file = URL.documentsDirectory.appendingPathComponent("recording.m4a") @State private var recorder: AVAudioRecorder? @State private var player: AVAudioPlayer? @State private var running = false @State private var interrupted = false @State private var status = "Ready" var body: some View { VStack(spacing: 24) { Button("Start") { Task { await start() } }.disabled(running) Button("Stop") { running = false interrupted = false recorder?.stop() status = file.path }.disabled(!running) Button("Play") { do { player = try AVAudioPlayer(contentsOf: file) status = "Saved duration: \(player!.duration) seconds" player?.play() } catch { status = error.localizedDescription } }.disabled(running) Text(status) } .padding() .task { for await notification in NotificationCenter.default.notifications( named: AVAudioSession.didBecomeInactiveNotification, object: session ) { guard running, let context = notification.userInfo?[AVAudioSession.deactivationContextKey] as? AVAudioSession.DeactivationContext, context.source == .system else { continue } recorder?.pause() interrupted = true status = "Interrupted" } } .task { for await notification in NotificationCenter.default.notifications( named: AVAudioSession.resumptionRecommendationNotification, object: session ) { guard running, interrupted, let context = notification.userInfo?[AVAudioSession.resumptionContextKey] as? AVAudioSession.ResumptionContext else { continue } guard context.recommendation == .shouldResume else { interrupted = false continue } do { guard try await session.activate(), running, interrupted else { continue } status = "Resumed: \(recorder!.record())" interrupted = false } catch { status = error.localizedDescription } } } } private func start() async { guard await AVAudioApplication.requestRecordPermission() else { return } do { try session.setCategory(.playAndRecord) guard try await session.activate() else { return } recorder = try AVAudioRecorder(url: file, settings: [ AVFormatIDKey: kAudioFormatMPEG4AAC_HE, AVSampleRateKey: 44_100, AVEncoderBitRateKey: 32_000, AVNumberOfChannelsKey: 2 ]) running = recorder!.record() status = "Recording: \(running)" } catch { status = error.localizedDescription } } }
Replies
0
Boosts
0
Views
451
Activity
1w
iPhone Duo: Does CameraCaptureAccessory accept touch, what counts as an active camera session, and can both front cameras run at once?
After watching "Leverage multiple displays and scenes on iPhone Duo" and "Build a great camera experience for iPhone Duo", we have four questions about the outer display while the device is open: Does a view hosted in CameraCaptureAccessory receive touch input on the outer display, or is it display-only? The accessory is described as available when the app "has an active camera session." Does that require a running preview layer or an attached output (photo, movie or data), or is a running AVCaptureSession with an input enough? Is a session whose main purpose is to host the accessory a supported use? Can AVCaptureMultiCamSession run the inner (under-display) front camera and the outer front camera at the same time on iPhone Duo? If so, which device-type combinations are supported, and are there thermal or power limits? Is CameraCaptureAccessory the only way for an app to show content on the outer display while the inner display is live, or are scene accessories that don't depend on the camera planned? Our use case is not photography. We're exploring a companion view for a second person sitting across from the device owner, so we want to understand whether the camera accessory is the right tool before designing around it.
Replies
5
Boosts
1
Views
352
Activity
2w
Why does Apple prefer Quad Bayer over traditional Bayer sensors?
Hi everyone, I've been reading about iPhone camera sensors recently, and there's something I've been wondering about. Apple has explained the benefits of Quad Bayer, especially pixel binning for low-light photography and high-resolution capture. But why not use a traditional Bayer sensor with Dual Pixel PDAF instead? As far as I understand, a native Bayer arrangement could potentially preserve more fine detail without requiring remosaicing. And if both sensors have the same size and pixel count, couldn't the Bayer sensor also produce a cleaner 12MP image by downsampling? I'm also curious about the microlens design. Some sensors use one microlens shared by four pixels for phase detection. Wouldn't individual microlenses potentially offer better spatial resolution? So my main question is: What hardware-level advantages make Quad Bayer worth choosing over traditional Bayer, despite the additional image reconstruction involved? Is it mainly about read noise, analog binning, power efficiency, autofocus, or something else? I'm not a camera engineer, just someone interested in understanding the technology behind iPhone cameras. I'd really appreciate any technical insight. Thanks!
Replies
0
Boosts
0
Views
108
Activity
2w
Feature request: level metering for MusicKit playback (FB25234486)
Hi everyone, I'm building a tvOS app that shows analog VU meters for the music playing, in the style of vintage hi-fi amplifiers and tape decks. It works well with local and network files played through AVAudioEngine, where I can tap the signal and measure real levels. With Apple Music, though, there's no way to get even basic level information. The content is DRM-protected, SystemMusicPlayer and ApplicationMusicPlayer render audio outside the app's audio graph, and MusicKit has no metering API. My current workaround is to analyze the song's 30-second preview and replay its level envelope in a loop. It looks plausible, but it isn't in sync with the actual music. I filed FB25234486 asking for a read-only metering API on MusicPlayer, similar to what AVAudioPlayer already offers: let rms = musicPlayer.averagePower(forChannel: 0) // dBFS let peak = musicPlayer.peakPower(forChannel: 0) // dBFS This wouldn't expose any audio samples: just one RMS and one peak value per channel, at a low, capped rate (around 30 Hz). Those numbers can't be turned back into audio, so content protection wouldn't be weakened, and the values could be computed inside the system's playback process. It would make a number of use cases possible: VU and peak meters, audio-reactive visualizers, ambient lighting synced to the music (HomeKit, smart lights), loudness displays, and accessibility features such as visual feedback of the music for hard-of-hearing users. If you need this too, please file your own feedback and mention FB25234486. Duplicate reports help Apple gauge demand. And if anyone has found a supported way to get playback levels from MusicKit today, I'd love to hear about it. Thanks!
Replies
0
Boosts
0
Views
127
Activity
2w
Ultra Wide camera: frame-to-frame jitter below 2.0x and sharpness step at 2.0x during native zoom ramp (iPhone 17 Pro, 4K/60)
I am developing CineZoom, part of CameraControl, a remote camera-direction system (macOS app controlling an iPhone capture engine). Its core requirement is a smooth, repeatable cinematic zoom. On iPhone 17 Pro (iOS 26.5.1), using the physical Back Ultra Wide camera at 3840x2160, 60 fps, a single native ramp(toVideoZoomFactor:withRate:) from 1.7x to 2.1x over 5 seconds shows a visible stutter. I isolated it with a minimal diagnostic app (no networking) and analysed four tripod recordings frame by frame, in two locations with different lighting. The results are consistent across all four: Jitter below 2.0x: while the zoom factor is below 2.0, the image centre shifts about 2.8 px RMS between consecutive frames (peaks ~5 px), alternating almost every frame (26-28 Hz). Above 2.0x, during the same ramp at the same speed, it drops to 1.2-1.5 px. Sharpness step at 2.0x: centre-crop sharpness drops by 23-31% within a few frames exactly when the zoom factor crosses 2.0. There is no consistent effect at the reported secondaryNativeResolutionZoomFactors value (1.7838). The zoom command path itself is correct: the zoom measured from the image matches videoZoomFactor throughout, with no dropped or duplicated frames. Questions: Is this expected behaviour for the Ultra Wide camera in this zoom range? Is the jitter related to video stabilization or to how the sensor crop is positioned in this readout mode? Is there a public API that reports or controls the zoom factor at which this transition happens, so an app can plan around it? What configuration do you recommend for a smooth continuous video zoom across 1.7x-2.1x on this camera (format, stabilization mode, constituent-camera switching)? Are there existing APIs that would let an app detect or compensate this (per-frame stabilization/crop metadata, consistent processing during zoom, or a reduced-processing video stream)? I have a full technical report, per-frame measurements, charts and the original recordings, and I am happy to share them. Thank you!
Replies
1
Boosts
0
Views
694
Activity
2w
AirPods camera-remote events for an external-camera controller
We are developing CameraDirector, an iPhone application that controls external professional cameras through documented vendor APIs and reads back native recording state. We want an AirPods camera-remote press to request record start/stop on the selected external camera, then provide audible confirmation only after the camera reports its state. In external-camera control mode, the app would be in the foreground but would not be capturing video from an iPhone camera. The WWDC25 session 'Enhancing your camera experience with capture controls' states that AVCaptureEventInteraction (or SwiftUI's onCameraCaptureEvent) requires an active AVCaptureSession and does not receive events when the app is backgrounded: https://developer.apple.com/videos/play/wwdc2025/253/ Is there a supported configuration for receiving AirPods capture events in this external-camera mode without running a local iPhone camera capture? Does an active audio-only AVCaptureSession qualify, or does event delivery require an active video-camera input? If this is outside the intended API contract, is there another supported input path for this use case, or should we file an enhancement request? We have not yet established an on-device AirPods result for this configuration. We would appreciate guidance on the intended architecture or a relevant sample and can provide a focused reproducer for an implementation issue. Related capture-event implementation discussion: https://developer.apple.com/forums/thread/740616 Thank you, Eddie DaRoza ACTION Cinema Technologies
Replies
1
Boosts
0
Views
735
Activity
2w
AVSpeechSynthesizer pulls words out of thin air.
Hi, I'm working on a project that uses the AVSpeechSynthesizer and AVSpeechUtterance. I discovered by chance that the AVSpeechSynthesizer automatically completes some words instead of just outputting what it's supposed to. These are abbreviations for days of the week or months, but not all of them. I don't want either of them automatically completed, but only the specified text. The completion transcends languages. I have written a short example program for demonstration purposes. import SwiftUI import AVFoundation import Foundation let synthesizer: AVSpeechSynthesizer = AVSpeechSynthesizer() struct ContentView: View { var body: some View { VStack { Button { utter("mon") } label: { Text("mon") } .buttonStyle(.borderedProminent) Button { utter("tue") } label: { Text("tue") } .buttonStyle(.borderedProminent) Button { utter("thu") } label: { Text("thu") } .buttonStyle(.borderedProminent) Button { utter("feb") } label: { Text("feb") } .buttonStyle(.borderedProminent) Button { utter("feb", lang: "de-DE") } label: { Text("feb DE") } .buttonStyle(.borderedProminent) Button { utter("wed") } label: { Text("wed") } .buttonStyle(.borderedProminent) } .padding() } private func utter(_ text: String, lang: String = "en-US") { let utterance = AVSpeechUtterance(string: text) let voice = AVSpeechSynthesisVoice(language: lang) utterance.voice = voice synthesizer.speak(utterance) } } #Preview { ContentView() } Thank you Christian
Replies
4
Boosts
0
Views
1.6k
Activity
3w