On iOS, SCStreamConfiguration.excludesCurrentProcessAudio appears to do nothing. Audio produced by our own process is captured at full level, so there is currently no way to capture device audio while excluding what our app is playing. Measurement (deterministic probe, 2026-08-08): Our app plays a 1 kHz tone at -4.9 dBFS RMS from its own process while capturing device audio with SCStreamConfiguration.capturesAudio = true and excludesCurrentProcessAudio = true. The configuration in force is read back from the stream and logged, so we know the flag is actually set. The captured audio contained our own tone at -4.9 dBFS peak. Separation = 0.0 dB. 1,573 audio sample buffers analyzed over 31.5 s (16 kHz mono, 50 buffers/s), zero analysis failures, and the run was reproduced twice about 3 hours apart with byte-identical verdicts. Environment for that run: iPhone 16 Pro (iPhone17,1), iOS 27.0, built with Xcode 27 beta 4 (27A5228h) against the iOS 27 SDK, installed directly from Xcode. Still reproducing on the current build: on iOS 27.0 (24A5418b) our shipping capture path still receives our own playback. In production we now have to cancel it ourselves - we align our playback buffer to the captured stream by cross-correlation and subtract it. The correlation between the captured signal and our own playback sits at |r| = 0.65-0.85 in the affected segments, i.e. the capture is dominated by a copy of our own output, exactly what excludesCurrentProcessAudio is supposed to remove. Doing this subtraction in-process costs real CPU and only works while we can hold a delay lock. Why this is blocking: RPSampleBufferType.audioApp is deprecated as of iOS 27 and the documentation points to ScreenCaptureKit as the replacement. With excludesCurrentProcessAudio non-functional there is no supported path on iOS to capture device audio while excluding one's own process. Our app is a real-time dubbing app - it captures foreign-language audio, transcribes it, and plays back a translated voice - so our own output re-entering the capture is fed straight back into transcription and corrupts the session. My questions: (1) Is excludesCurrentProcessAudio expected to be functional on iOS 27, or is it macOS-only in practice? The documentation does not mark it as unavailable on iOS. (2) If it is expected to work, is there anything the app must do besides setting it on the SCStreamConfiguration used to start the stream? (3) If it is not going to work on iOS, what is the supported way to exclude the current process's audio from a ScreenCaptureKit capture, now that RPSampleBufferType.audioApp is deprecated? Filed as FB24170972 on 2026-08-08, with the full JSON event logs from both probe runs attached. There has been no response on the Feedback, which is why I am raising it here.

On iOS, ScreenCaptureKit terminates an SCStream when the app is backgrounded unless the app declares UIBackgroundModes: screen-capture. The delegate reports SCStreamError code -3824 (SCStreamError.Code.missingBackgroundMode). Adding screen-capture to UIBackgroundModes - the value Apple's own iOS 27 ScreenCaptureKit sample code declares - makes App Store Connect reject the upload: error: exportArchive Invalid Info.plist value. The Info.plist key UIBackgroundModes contains an invalid value: 'screen-capture'. So the app cannot be distributed at all, not even to TestFlight internal testers. We have ruled out simply keeping the process alive: running a continuous silent AVAudioEngine under the audio background mode keeps the app running but does not prevent -3824, which is consistent with the framework performing an explicit background-mode check rather than the stream dying from process suspension. One observation that may explain why this has gone unnoticed: Apple's sample is installed directly from Xcode and therefore never passes through App Store Connect's upload validation, so a mismatch between the sample's Info.plist and App Store Connect's allowlist would not be visible when testing the sample internally. My question is which of these three describes the actual status of screen-capture for third-party iOS apps: (1) App Store Connect's allowlist has not yet been updated for iOS 27, and this will resolve on its own. (2) screen-capture requires an entitlement or a specially provisioned profile we have not requested. If so, which one, and how is it requested? (3) screen-capture is restricted to Apple's own or system applications and is not available to third-party App Store apps by design. I could not find screen-capture documented on the general UIBackgroundModes page, which is why I cannot tell these apart. Filed as FB24169650. Device: iPhone 16 Pro, iOS 27.0 Built against the iOS 27 SDK, deployment target 17.0.

Product area macOS / Privacy & Security / ScreenCaptureKit / Core Graphics Environment macOS 27 Beta 4 (build: fill in your exact build number, e.g. 27A5xxx) Xcode 26.5 / SDK 260500 (adjust to match what you actually built with) App holds the com.apple.developer.persistent-content-capture entitlement (approved via Apple's request form), targeting macOS 14.4+ Summary Our app is a remote-support/remote-control tool (screen viewing + control), comparable to VNC-style products. On macOS 27, we've found that System Settings > Privacy & Security now shows a "Remote Desktop" entry that is distinct from "Screen & System Audio Recording" — granting one does not affect the other. We need a way to check, at any time, whether our app currently has "Remote Desktop" authorization, without causing the system to show a permission-request alert as a side effect. We have not found a documented, public API that does this. What we've tried CGPreflightScreenCaptureAccess() Confirmed via a controlled test on-device: granting only "Remote Desktop" leaves this API returning false; granting only "Screen & System Audio Recording" makes it return true. So this API appears to reflect kTCCServiceScreenCapture only, and does not reflect the "Remote Desktop" permission at all. ScreenCaptureKit (SCShareableContent, e.g. via a refreshAvailableContentWithCompletionHandler:-style call) This call does appear to interact with the "Remote Desktop" permission — but calling it triggers a real system consent alert every time we call it, even when we only intend to read the current status, not request it. This makes it unusable for passive/background status polling (e.g. to decide what to show in our own onboarding UI without surprising the user with an OS-level prompt). We are intentionally not reading /Library/Application Support/com.apple.TCC/TCC.db directly — we understand this is a private, undocumented database and want a supported API instead. Sample code illustrating both attempts // Attempt 1: CGPreflightScreenCaptureAccess — does not reflect Remote Desktop grant BOOL preflightResult = CGPreflightScreenCaptureAccess(); // preflightResult stays NO even after the user grants "Remote Desktop" in // System Settings > Privacy & Security > Remote Desktop. // It correctly flips to YES only when "Screen & System Audio Recording" is granted. // Attempt 2: ScreenCaptureKit-based check — reflects it, but prompts every time SCShareableContent... // (via our wrapper) refreshAvailableContentWithCompletionHandler: // This call appears to influence/query the Remote Desktop TCC entry, but the OS // shows a permission alert as a side effect of the call itself, even when we only // want to read the current authorization state. Question Is there a public, documented API equivalent to CGPreflightScreenCaptureAccess() — i.e., a read-only, non-prompting status check — for the new "Remote Desktop" privacy category introduced around macOS 26/27? Is com.apple.developer.persistent-content-capture actually the entitlement that governs this new "Remote Desktop" category, or is it unrelated? Apple's own documentation describes this entitlement purely in terms of "persistent access to screen capture" for VNC apps, with no mention of a distinct "Remote Desktop" permission surface — we'd like to confirm whether that description is still accurate on macOS 26/27, or whether the underlying TCC service (kTCCServiceRemoteDesktop, which we found via TCC.db schema inspection only, not public docs) has been intentionally split out. If no such API exists yet, is this planned, and is there a recommended interim approach for apps that need to know this state before deciding whether to show their own onboarding/permission UI?

I am using SCScreenshotManager.captureImage to capture image of an app window. i have the sleep settings that sleeps the display but not the system. when the display sleeps taking screenshot using SCScreenshotManager.captureImage does not work. Is there a way i can take screenshot of window when the display sleeps? Thanks

When using the built-in macOS screen recording feature, the recording stops automatically after approximately 10–15 minutes without any warning or error message. No manual stop action is performed. The recording simply ends silently. The same issue also occurs when using ScreenCaptureKit in a custom application, which suggests this may be a system-level issue related to screen capture rather than an app-specific problem. This issue is reproducible and happens consistently after running for a period of time.

When using the built-in macOS screen recording feature, the recording stops automatically after approximately 10–15 minutes without any warning or error message. No manual stop action is performed. The recording simply ends silently. The same issue also occurs when using ScreenCaptureKit in a custom application, which suggests this may be a system-level issue related to screen capture rather than an app-specific problem. This issue is reproducible and happens consistently after running for a period of time.

Hello Apple Developer Support, I am reaching out for technical guidance regarding screen capture protection behavior on macOS. We are building a desktop application using Flutter running on macOS, and we have implemented native Swift code inside the macOS Runner in order to protect sensitive content from screen recording and screen sharing. Our current implementation relies on native window-level protection and display state handling from Swift, while the main UI remains rendered by Flutter. The main challenge we are facing is the following: we need to keep a strong native anti-recording protection on macOS the application is heavily used with AirPlay and screen mirroring currently, AirPlay / mirroring is often interpreted by the system similarly to screen capture or screen recording this causes our protected content to be replaced by a gray or blank area even during legitimate AirPlay usage In practice, we would like to allow: AirPlay legitimate external display / mirroring usage while still preventing: screen recording screen sharing unauthorized screen capture We would like to know whether Apple recommends an official supported approach for this use case, preferably using public APIs. More specifically: Is there an officially supported way on macOS to distinguish AirPlay mirroring from screen recording / screen sharing? Is "NSWindow.sharingType" the recommended public API for this scenario? Is there a recommended approach when the UI surface is rendered through Flutter / Metal? Are there any best practices with ScreenCaptureKit for protecting content without affecting AirPlay? We understand that some lower-level APIs may not be officially supported, so we would greatly appreciate guidance toward a public and future-proof implementation path. Thank you very much for your time and support. Best regards, Tony

Bug Report: ScreenCaptureKit System Audio Capture Crashes with EXC_BAD_ACCESS Summary When using ScreenCaptureKit to capture system audio for extended periods, the application crashes with EXC_BAD_ACCESS in Swift's error handling runtime. The crash occurs in swift_getErrorValue when trying to process an error from the SCStream delegate method didStopWithError. This appears to be a framework-level issue in ScreenCaptureKit or its underlying ReplayKit implementation. Environment macOS Sonoma 14.6.1 Swift 5.8 ScreenCaptureKit framework Detailed Description Our application captures system audio using ScreenCaptureKit's audio capture capabilities. After successfully capturing for several minutes (typically after 3-4 segments of 60-second recordings), the application crashes with an EXC_BAD_ACCESS error. The crash happens when the Swift runtime attempts to process an error in the SCStreamDelegate.stream(_:didStopWithError:) method. The crash consistently occurs in swift_getErrorValue when attempting to access the class of what appears to be a null object. This suggests that the error being passed from the system framework to our delegate method is malformed or contains invalid memory. Steps to Reproduce Create an SCStream with audio capture enabled Add audio output to the stream Start capture and write audio data to disk Allow the capture to run for several minutes (3-5 minutes typically triggers the issue) The app will crash with EXC_BAD_ACCESS in swift_getErrorValue Code Sample func stream(_ stream: SCStream, didStopWithError error: Error) { print("Stream stopped with error: \(error)") // Crash occurs before this line executes } func stream(_ stream: SCStream, didOutputSampleBuffer sampleBuffer: CMSampleBuffer, of type: SCStreamOutputType) { guard type == .audio, sampleBuffer.isValid else { return } // Process audio data... } Expected Behavior The error should be properly propagated to the delegate method, allowing for graceful error handling and recovery. Actual Behavior The application crashes with EXC_BAD_ACCESS when the Swift runtime attempts to process the error in swift_getErrorValue. Crash Log Details Thread #35, queue = 'com.apple.NSXPCConnection.m-user.com.apple.replayd', stop reason = EXC_BAD_ACCESS (code=1, address=0x0) frame #0: 0x0000000194c3088c libswiftCore.dylib`swift::_swift_getClass(void const*) + 8 frame #1: 0x0000000194c30104 libswiftCore.dylib`swift_getErrorValue + 40 frame #2: 0x00000001057fba30 shadow`NewScreenCaptureService.stream(stream=0x0000600002de6700, error=Swift.Error @ 0x000000016b7b5e30) at NEW+ScreenCaptureService.swift:365:15 frame #3: 0x00000001057fc050 shadow`@objc NewScreenCaptureService.stream(_:didStopWithError:) at <compiler-generated>:0 frame #4: 0x0000000219ec5ca0 ScreenCaptureKit`-[SCStreamManager stream:didStopWithError:] + 456 frame #5: 0x00000001ca68a5cc ReplayKit`-[RPScreenRecorder stream:didStopWithError:] + 84 frame #6: 0x00000001ca696ff8 ReplayKit`-[RPDaemonProxy stream:didStopWithError:] + 224 Printing description of stream._streamQueue: error: ObjectiveC.id:4294967281:18: note: 'id' has been explicitly marked unavailable here public typealias id = AnyObject ^ error: /var/folders/v4/3xg1hmp93gjd8_xlzmryf_wm0000gn/T/expr23-dfa421..cpp:1:65: 'id' is unavailable in Swift: 'id' is not available in Swift; use 'Any' Swift._DebuggerSupport.stringForPrintObject(Swift.UnsafePointer<id>(bitPattern: 0x104ae08c0)!.pointee) ^~ ObjectiveC.id:2:18: note: 'id' has been explicitly marked unavailable here public typealias id = AnyObject ^ warning: /var/folders/v4/3xg1hmp93gjd8_xlzmryf_wm0000gn/T/expr23-dfa421..cpp:5:7: initialization of variable '$__lldb_error_result' was never used; consider replacing with assignment to '_' or removing it var $__lldb_error_result = __lldb_tmp_error ~~~~^~~~~~~~~~~~~~~~~~~~ _ Before the crash, we observed this error message in the console: [ERROR] *****SCStream*****RemoteAudioQueueOperationHandlerWithError:1015 Error received from the remote queue -16665 Additional Context The issue occurs consistently after approximately 3-4 successful audio segment recordings of 60 seconds each Commenting out custom segment rotation logic does not prevent the crash The crash involves XPC communication with Apple's ReplayKit daemon The error appears to be corrupted or malformed when crossing the XPC boundary Workarounds Attempted Added proper thread safety for all published properties using DispatchQueue.main.async Implemented more robust error handling in the delegate methods None of these approaches prevented the crash since it occurs at the Swift runtime level before our code executes. Impact This issue prevents reliable long-duration audio capture using ScreenCaptureKit. This bug significantly limits the usefulness of ScreenCaptureKit for any application requiring continuous system audio capture for more than a few minutes. Perhaps this issue might be related to a macOS bug where the system dialog indicates that the screen is being shared, even though nothing is actually being shared. Moreover, when attempting to stop sharing, nothing happens.

Hi, I'm new to AVAudioEngine(and macOS programming in general). I'm trying to mix microphone audio with ScreenCaptureKit audio using AVAudioEngine without playing it back. I've created a AVAudioPlayerNode and scheduling buffers in my SCStream handler: playerNode.scheduleBuffer(samples) and have connected the playerNode to the mainMixerNode. audioEngine.connect(audioEngine.inputNode, to: audioEngine.mainMixerNode, format: micFormat) audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: format) The problem is that mainMixerNode plays the audio to the speaker creating a feedback loop. How can I prevent the mixer output from being played back. Also: Is this the best way of mixing microphone input with some other input? I ran into AVAudioEngine's manual rendering mode, which seems like the way to go for mixing audio without playing it back. However, I couldn't figure out how to connect microphone input to the AVAudioEngine in manual rendering mode?

Precondition: In system settings, scale the pointer size up to the max. Our SCScreenshotManager code currently works in macOS 15 and earlier to capture the cursor at it's larger size, but broke in one of the minor releases of macOS Tahoe. The error it produces now is "Failed to start stream due to audio/video capture failure". This only seems to happen with the cursor window, not any others. Another way to get the cursor is with https://developer.apple.com/documentation/appkit/nscursor/currentsystem, but that is now deprecated, which makes me think the capture of the cursor is being blocked deliberately. We see this as a critical loss of functionality for our apps, and could use guidance on what to use instead.

Hello everyone, I'm working on a screen recording app using ScreenCaptureKit and I've hit a strange issue. My app records the screen to an .mp4 file, and everything works perfectly until the .captureMicrophone is false In this case, I get a valid, playable .mp4 file. However, as soon as I try to enable the microphone by setting streamConfig.captureMicrophone = true, the recording seems to work, but the final .mp4 file is corrupted and cannot be played by QuickTime or any other player. This happens whether capturesAudio (app audio) is on or off. I've already added the "Privacy - Microphone Usage Description" (NSMicrophoneUsageDescription) to my Info.plist, so I don't think it's a permissions problem. I have my logic split into a ScreenRecorder class that manages state and a CaptureEngine that handles the SCStream. Here is how I'm configuring my SCStream: ScreenRecorder.swift // This is my main SCStreamConfiguration private var streamConfiguration: SCStreamConfiguration { var streamConfig = SCStreamConfiguration() // ... other HDR/preset config ... // These are the problem properties streamConfig.capturesAudio = isAudioCaptureEnabled streamConfig.captureMicrophone = isMicCaptureEnabled // breaks it if true streamConfig.excludesCurrentProcessAudio = false streamConfig.showsCursor = false if let region = selectedRegion, let display = currentDisplay { // My region/frame logic (works fine) let regionWidth = Int(region.frame.width) let regionHeight = Int(region.frame.height) streamConfig.width = regionWidth * scaleFactor streamConfig.height = regionHeight * scaleFactor // ... (sourceRect logic) ... } streamConfig.pixelFormat = kCVPixelFormatType_32BGRA streamConfig.colorSpaceName = CGColorSpace.sRGB streamConfig.minimumFrameInterval = CMTime(value: 1, timescale: 60) return streamConfig } And here is how I'm setting up the SCRecordingOutput that writes the file: ScreenRecorder.swift private func initRecordingOutput(for region: ScreenPickerManager.SelectedRegion) throws { let screeRecordingOutputURL = try RecordingWorkspace.createScreenRecordingVideoFile( in: workspaceURL, sessionIndex: sessionIndex ) let recordingConfiguration = SCRecordingOutputConfiguration() recordingConfiguration.outputURL = screeRecordingOutputURL recordingConfiguration.outputFileType = .mp4 recordingConfiguration.videoCodecType = .hevc let recordingOutput = SCRecordingOutput(configuration: recordingConfiguration, delegate: self) self.recordingOutput = recordingOutput } Finally, my CaptureEngine adds these to the SCStream: CaptureEngine.swift class CaptureEngine: NSObject, @unchecked Sendable { private(set) var stream: SCStream? private var streamOutput: CaptureEngineStreamOutput? // ... (dispatch queues) ... func startCapture(configuration: SCStreamConfiguration, filter: SCContentFilter, recordingOutput: SCRecordingOutput) async throws { let streamOutput = CaptureEngineStreamOutput() self.streamOutput = streamOutput do { stream = SCStream(filter: filter, configuration: configuration, delegate: streamOutput) // Add outputs for raw buffers (not used for file recording) try stream?.addStreamOutput(streamOutput, type: .screen, sampleHandlerQueue: videoSampleBufferQueue) try stream?.addStreamOutput(streamOutput, type: .audio, sampleHandlerQueue: audioSampleBufferQueue) try stream?.addStreamOutput(streamOutput, type: .microphone, sampleHandlerQueue: micSampleBufferQueue) // Add the file recording output try stream?.addRecordingOutput(recordingOutput) try await stream?.startCapture() } catch { logger.error("Failed to start capture: \(error.localizedDescription)") throw error } } // ... (stopCapture, etc.) ... } When I had the .captureMicrophone value to be false, I get a perfect .mp4 video playable everywhere, however, when its true, I am getting corrupted video which doesn't play at all :-

We built an open-source macOS menu bar app that turns speech into text and pastes it into the active app — using SpeechAnalyzer for on-device transcription, ScreenCaptureKit + Vision for screen-aware context, and FluidAudio for speaker diarization in meeting mode. Here's what we learned shipping it on macOS 26. GitHub: github.com/Marvinngg/ambient-voice Architecture The app has two modes: hotkey dictation (press to talk, release to inject) and meeting recording (continuous transcription with a floating panel). Dictation Mode Audio capture uses AVCaptureSession (more on why below). The captured audio feeds into SpeechAnalyzer via an AsyncStream: let transcriber = SpeechTranscriber( locale: locale, transcriptionOptions: [], reportingOptions: [.volatileResults, .alternativeTranscriptions], attributeOptions: [.audioTimeRange, .transcriptionConfidence] ) let analyzer = SpeechAnalyzer(modules: [transcriber]) let (inputSequence, inputBuilder) = AsyncStream.makeStream() try await analyzer.start(inputSequence: inputSequence) While recording, we capture a screenshot of the focused window using ScreenCaptureKit, run Vision OCR (VNRecognizeTextRequest), extract keywords, and inject them into SpeechAnalyzer as contextual bias: let context = AnalysisContext() context.contextualStrings[.general] = ocrKeywords try await analyzer.setContext(context) This improves accuracy for technical terms and proper nouns visible on screen. If your screen shows "SpeechAnalyzer", saying it out loud is more likely to be transcribed correctly. After transcription, an optional L2 step sends the text through a local LLM (ollama) for spoken-to-written cleanup, then CGEvent simulates Cmd+V to paste into the active app. Meeting Mode Meeting mode forks the same audio stream to two consumers: SpeechAnalyzer — real-time streaming transcription, displayed in a floating NSPanel FluidAudio buffer — accumulates 16kHz Float32 mono samples for batch speaker diarization after recording stops When the user ends the meeting, FluidAudio's performCompleteDiarization() runs on the accumulated audio. We align transcription segments with speaker segments using audioTimeRange overlap matching — each transcription segment gets assigned the speaker ID with the most time overlap. Results export to Markdown. Pitfalls We Hit on macOS 26 1. AVAudioEngine installTap doesn't fire with Bluetooth devices We started with AVAudioEngine.inputNode.installTap() for audio capture. It worked fine with built-in mics but the tap callback never fired with Bluetooth devices (tested with vivo TWS 4 Hi-Fi). Fix: switched to AVCaptureSession. The delegate callback captureOutput(_:didOutput:from:) fires reliably regardless of audio device. The tradeoff is you get CMSampleBuffer instead of AVAudioPCMBuffer, so you need a conversion step. 2. NSEvent addGlobalMonitorForEvents crashes Our global hotkey listener used NSEvent.addGlobalMonitorForEvents. On macOS 26, this crashes with a Bus error inside GlobalObserverHandler — appears to be a Swift actor runtime issue. Fix: switched to CGEventTap. Works reliably, but the callback runs on a CFRunLoop context, which Swift doesn't recognize as MainActor. 3. CGEventTap callbacks aren't on MainActor If your CGEventTap callback touches any @MainActor state, you'll get concurrency violations. The callback runs on whatever thread owns the CFRunLoop. Fix: bridge with DispatchQueue.main.async {} inside the tap callback before touching any MainActor state. 4. CGPreflightScreenCaptureAccess doesn't request permission We used CGPreflightScreenCaptureAccess() as a guard before calling ScreenCaptureKit. If it returned false, we'd bail out. The problem: this function only checks — it never triggers macOS to add your app to the Screen Recording permission list. Chicken-and-egg: you can't get permission because you never ask for it. Fix: call CGRequestScreenCaptureAccess() at app startup. This adds your app to System Settings → Screen Recording. Then let ScreenCaptureKit calls proceed without the preflight guard — SCShareableContent will also trigger the permission prompt on first use. 5. Ad-hoc signing breaks TCC permissions on every rebuild During development, codesign --sign - (ad-hoc) generates a different code directory hash on every build. macOS TCC tracks permissions by this hash, so every rebuild = new app identity = all permissions reset. Fix: sign with a stable certificate. If you have an Apple Development certificate, use that. The TeamIdentifier stays constant across rebuilds, so TCC permissions persist. We also discovered that launching via open WE.app (LaunchServices) instead of directly executing the binary is required — otherwise macOS attributes TCC permissions to Terminal, not your app. Benchmarks We ran end-to-end benchmarks on public datasets (Mac Mini M4 16GB, macOS 26): Transcription (SpeechAnalyzer, AliMeeting Chinese): • Near-field CER 34% (excluding outliers ~25%) • Far-field CER 40% (single channel, no beamforming, >30% overlap) • Processing speed 74-89x real-time Speaker diarization (FluidAudio offline): • AMI English 16 meetings: avg DER 23.2% (collar=0.25s, ignoreOverlap=True) • AliMeeting Chinese 8 meetings: DER 48.5% (including overlap regions) • Memory: RSS ~500MB, peak 730-930MB Full evaluation methodology, scripts, and raw results are in the repo. Open Source The project is MIT licensed: github.com/Marvinngg/ambient-voice It includes the macOS client (Swift 6.2, SPM), server-side distillation/training scripts (Python), and a complete evaluation framework with reproducible benchmarks. Feedback and contributions welcome.

We built an open-source macOS menu bar app that turns speech into text and pastes it into the active app — using SpeechAnalyzer for on-device transcription, ScreenCaptureKit + Vision for screen-aware context, and FluidAudio for speaker diarization in meeting mode. Here's what we learned shipping it on macOS 26. GitHub: github.com/Marvinngg/ambient-voice Architecture The app has two modes: hotkey dictation (press to talk, release to inject) and meeting recording (continuous transcription with a floating panel). Dictation Mode Audio capture uses AVCaptureSession (more on why below). The captured audio feeds into SpeechAnalyzer via an AsyncStream: let transcriber = SpeechTranscriber( locale: locale, transcriptionOptions: [], reportingOptions: [.volatileResults, .alternativeTranscriptions], attributeOptions: [.audioTimeRange, .transcriptionConfidence] ) let analyzer = SpeechAnalyzer(modules: [transcriber]) let (inputSequence, inputBuilder) = AsyncStream.makeStream() try await analyzer.start(inputSequence: inputSequence) While recording, we capture a screenshot of the focused window using ScreenCaptureKit, run Vision OCR (VNRecognizeTextRequest), extract keywords, and inject them into SpeechAnalyzer as contextual bias: let context = AnalysisContext() context.contextualStrings[.general] = ocrKeywords try await analyzer.setContext(context) This improves accuracy for technical terms and proper nouns visible on screen. If your screen shows "SpeechAnalyzer", saying it out loud is more likely to be transcribed correctly. After transcription, an optional L2 step sends the text through a local LLM (ollama) for spoken-to-written cleanup, then CGEvent simulates Cmd+V to paste into the active app. Meeting Mode Meeting mode forks the same audio stream to two consumers: SpeechAnalyzer — real-time streaming transcription, displayed in a floating NSPanel FluidAudio buffer — accumulates 16kHz Float32 mono samples for batch speaker diarization after recording stops When the user ends the meeting, FluidAudio's performCompleteDiarization() runs on the accumulated audio. We align transcription segments with speaker segments using audioTimeRange overlap matching — each transcription segment gets assigned the speaker ID with the most time overlap. Results export to Markdown. Pitfalls We Hit on macOS 26 1. AVAudioEngine installTap doesn't fire with Bluetooth devices We started with AVAudioEngine.inputNode.installTap() for audio capture. It worked fine with built-in mics but the tap callback never fired with Bluetooth devices (tested with vivo TWS 4 Hi-Fi). Fix: switched to AVCaptureSession. The delegate callback captureOutput(_:didOutput:from:) fires reliably regardless of audio device. The tradeoff is you get CMSampleBuffer instead of AVAudioPCMBuffer, so you need a conversion step. 2. NSEvent addGlobalMonitorForEvents crashes Our global hotkey listener used NSEvent.addGlobalMonitorForEvents. On macOS 26, this crashes with a Bus error inside GlobalObserverHandler — appears to be a Swift actor runtime issue. Fix: switched to CGEventTap. Works reliably, but the callback runs on a CFRunLoop context, which Swift doesn't recognize as MainActor. 3. CGEventTap callbacks aren't on MainActor If your CGEventTap callback touches any @MainActor state, you'll get concurrency violations. The callback runs on whatever thread owns the CFRunLoop. Fix: bridge with DispatchQueue.main.async {} inside the tap callback before touching any MainActor state. 4. CGPreflightScreenCaptureAccess doesn't request permission We used CGPreflightScreenCaptureAccess() as a guard before calling ScreenCaptureKit. If it returned false, we'd bail out. The problem: this function only checks — it never triggers macOS to add your app to the Screen Recording permission list. Chicken-and-egg: you can't get permission because you never ask for it. Fix: call CGRequestScreenCaptureAccess() at app startup. This adds your app to System Settings → Screen Recording. Then let ScreenCaptureKit calls proceed without the preflight guard — SCShareableContent will also trigger the permission prompt on first use. 5. Ad-hoc signing breaks TCC permissions on every rebuild During development, codesign --sign - (ad-hoc) generates a different code directory hash on every build. macOS TCC tracks permissions by this hash, so every rebuild = new app identity = all permissions reset. Fix: sign with a stable certificate. If you have an Apple Development certificate, use that. The TeamIdentifier stays constant across rebuilds, so TCC permissions persist. We also discovered that launching via open WE.app (LaunchServices) instead of directly executing the binary is required — otherwise macOS attributes TCC permissions to Terminal, not your app. Benchmarks We ran end-to-end benchmarks on public datasets (Mac Mini M4 16GB, macOS 26): Transcription (SpeechAnalyzer, AliMeeting Chinese): • Near-field CER 34% (excluding outliers ~25%) • Far-field CER 40% (single channel, no beamforming, >30% overlap) • Processing speed 74-89x real-time Speaker diarization (FluidAudio offline): • AMI English 16 meetings: avg DER 23.2% (collar=0.25s, ignoreOverlap=True) • AliMeeting Chinese 8 meetings: DER 48.5% (including overlap regions) • Memory: RSS ~500MB, peak 730-930MB Full evaluation methodology, scripts, and raw results are in the repo. Open Source The project is MIT licensed: github.com/Marvinngg/ambient-voice It includes the macOS client (Swift 6.2, SPM), server-side distillation/training scripts (Python), and a complete evaluation framework with reproducible benchmarks. Feedback and contributions welcome.

Sharing a solution for a problem that took me a while to figure out. Problem: During development of a macOS 26 app that uses ScreenCaptureKit, the screen capture permissions were being reset after every build. Each time I compiled and ran the app from Xcode, I had to re-authorize screen capture in System Settings. CGPreflightScreenCaptureAccess() would return false even though I'd just granted permission minutes ago. Root cause: I was using ad-hoc code signing during development. macOS ties screen capture permissions to the app's code signing identity. With ad-hoc signing, the identity changes on every build, so the system treats each build as a "new" app. Solution: Switch to an Apple Development certificate for debug builds. In Xcode: Build Settings → Code Signing Identity → Debug → set to "Apple Development" Make sure your development team is selected After this change, the signing identity remains stable across builds, and screen capture permissions persist. This might be related to the broader issue discussed in this forum about ScreenCapture permissions disappearing — if other developers are seeing permissions vanish, it's worth checking whether the code signing identity is changing between sessions.

On Tahoe and earlier, ScreenCapture permissions can disappear and not return. Customers are having an issue with this disappearing and when our code executes CGRequestScreenCaptureAccess() nothing happens, the prompt does not appear. I can reproduce this by using the "-" button and removing the entry in the settings, then adding it back with the "+" button. CGPreflightScreenCaptureAccess() always returns the correct value but once the entry has been removed, CGRequestScreenCaptureAccess() requires a reboot before it will work again.

Two months ago I requested the subject entitlement. I'm still waiting for it to be added to our account. Who or how can I find out what going on with it. I have no correspondence from Apple yet saying it was denied and why. https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.persistent-content-capture?language=objc Thank you.

With the help of Claude and Codex, I've tried upgrading a screen magnification, capture, and pixel peeping app (SnoopX, from Stanford's software repository) to be HDR-aware. For images that are truly HDR (displayed in Apple Photos, Google Photos in Chrome, or Mac Preview), and on an HDR monitor with plenty of EDR headroom (Apple XDR display), I should see pixel values well above 1.0. Apple's Digital Color Meter does. However, my app does not. I think the code is doing all the right things with ScreenCaptureKit, but it never returns values above 1.0. Has anybody gotten this to work? Here is what Codex says about the code it helped me build in my app: • Set captureDynamicRange: We set config​.capture​Dynamic​Range = ​SCCapture​Dynamic​Range​HDRLocal​Display (when available). Note: Gemini’s SCCapture​Mode​HDRLocal​Display isn’t a real symbol — the actual API is SCCapture​Dynamic​Range. • Use preset: We try init​With​Preset: ​SCStream​Configuration​Preset​Capture​HDRStream​Local​Display, but your runtime crashes on init​With​Preset: (unrecognized selector), so we fall back to manual config. That means your system SDK headers mention the preset, but the runtime doesn’t implement it. • Pixel format: We explicitly set k​CVPixel​Format​Type​_64​RGBAHalf. Your logs show the stream buffer is 64RGBAHalf. • Enable HDR on display: EDR headroom logs show current=6.40, so HDR is active. • Color space: We set Extended Linear Display P3 for stream, and we forced the screenshot sampler to convert into extended‑linear P3 before reading. The screenshot itself reports Extended sRGB, but conversion doesn’t yield >1.0 either. Despite all of that, the OS is still delivering tone‑mapped values (max 1.0000) for both the stream and the screenshot path. So I don’t think there’s a missing knob here; it looks like a system‑level limitation on Tahoe for desktop capture.

We're troubleshooting SCK issues. They occur with a relatively small amount of sessions, but lack of context and/or ability to advise the customer on how they could make behavior more predictable and reliable is problematic. Generally, there is 2 distinct issues which may or may not have the same root cause: Failure to establish SCK session. Usually manifests within the app as SCShareableContent.getWithCompletionHandler call either never invoking the completion handler, or taking prohibitively long time (we usually give it 3-10 sec before giving up). In the system log it may look like this: (log omitted - suspecting it triggers the content filter) Note the 6 seconds delay to completion of fetchShareableContentWithOption (normally it's a 30-40ms operation). Sometime, we'd see the stream established, but some minutes (or even hours) into the recording we'd stop receiving frames. Both scenarios are likely to occur when the disk space is low, with reliable repro of the problem #2 at below 8gb of free space (in that case, we've seen replayd silently dropping the session, without ever notifying the client ... improving API could go a long way there). However, out of recent occurrences, while most have less than 100GB available, we've seen it on machines with as much as 500GB free. Unfortunately, it's almost never reproducible in dev environment, so we have to rely on diagnostics we're able to collect in the field -- which nothing obvious yet. I'd like to understand the root cause of both scenarios better and/or how what specific frameworks can cause these behaviors.

I have an app that records a 32 x 32 rect under the cursor as the user moves it around and it sends it to Flutter. It suffers from major lag. Instead of getting 30 fps, I get about 7 fps. That is, there are significant lags between screen grabs. This on an Intel Mac mini x64 with 15.7.3 and one display. flutter: NATIVE: ExplodedView framesIn=2 timeSinceStart=1115.7ms gapSinceLastFrame=838.8ms flutter: NATIVE: ExplodedView framesIn=4 timeSinceStart=1382.6ms gapSinceLastFrame=149.9ms flutter: NATIVE: ExplodedView framesIn=5 timeSinceStart=1511.0ms gapSinceLastFrame=128.4ms flutter: NATIVE: ExplodedView framesIn=7 timeSinceStart=1698.3ms gapSinceLastFrame=102.9ms flutter: NATIVE: ExplodedView STOP polling totalTime=4482.6ms framesIn=28 framesSent=28 acks=28 Here's a testable excerpt: import ScreenCaptureKit import CoreMedia import CoreVideo import QuartzCore final class Test: NSObject, SCStreamOutput, SCStreamDelegate { private let q = DispatchQueue(label: "cap.q") private var stream: SCStream? private var lastFrameAt: CFTimeInterval = 0 private var frames = 0 func start() { SCShareableContent.getExcludingDesktopWindows(false, onScreenWindowsOnly: true) { content, err in guard err == nil, let display = content?.displays.first else { print("shareableContent error: \(String(describing: err))"); return } let filter = SCContentFilter(display: display, excludingWindows: []) let config = SCStreamConfiguration() config.showsCursor = false config.queueDepth = 1 config.minimumFrameInterval = CMTime(value: 1, timescale: 30) config.pixelFormat = kCVPixelFormatType_32BGRA config.width = 32 config.height = 32 config.sourceRect = CGRect(x: 100, y: 100, width: 32, height: 32) let s = SCStream(filter: filter, configuration: config, delegate: self) try! s.addStreamOutput(self, type: .screen, sampleHandlerQueue: self.q) self.stream = s s.startCapture { startErr in print("startCapture err=\(String(describing: startErr))") } // Optional: move sourceRect at 30Hz (cursor-follow simulation) Timer.scheduledTimer(withTimeInterval: 1.0/30.0, repeats: true) { _ in let c2 = SCStreamConfiguration() c2.showsCursor = false c2.queueDepth = 1 c2.minimumFrameInterval = CMTime(value: 1, timescale: 30) c2.pixelFormat = kCVPixelFormatType_32BGRA c2.width = 32 c2.height = 32 let t = CACurrentMediaTime() c2.sourceRect = CGRect(x: 100 + (sin(t) * 50), y: 100, width: 32, height: 32) s.updateConfiguration(c2) { _ in } } } } func stream(_ stream: SCStream, didOutputSampleBuffer sb: CMSampleBuffer, of type: SCStreamOutputType) { guard type == .screen else { return } let now = CACurrentMediaTime() let gapMs = (lastFrameAt == 0) ? 0 : (now - lastFrameAt) * 1000 lastFrameAt = now frames += 1 if frames <= 10 || frames % 60 == 0 { print("frames=\(frames) gapMs=\(String(format: "%.1f", gapMs))") } } }

Summary I am investigating a change in macOS 26.1 (Tahoe) where plain (non-bundled) executables that request screen recording access no longer appear under: System Settings → Privacy & Security → Screen & System Audio Recording This behavior differs from macOS Sequoia, where these executables did appear in the list and could be managed through the UI. Tahoe still prompts for permission and still allows the executable to capture the screen once permission is granted, but the executable never shows up in the UI list. This breaks user expectations and removes UI-based permission management. To confirm the behavior, I created a small reproduction project with both: a plain executable, and an identical executable packaged inside an .app bundle. Only the bundled version appears in System Settings. Observed Behaviour 1. Plain Executable (from my reproduction project) When running a plain executable that captures the screen: macOS displays the normal screen-recording permission prompt. Before granting permission: screenshots show only the desktop background. After granting permission: screenshots capture the full display. The executable does not appear under “Screen & System Audio Recording”. Even when permission is granted manually (e.g., dragging the executable into the pane), the executable still does not appear, which prevents the user from modifying or revoking the permission through the UI. If the executable is launched from inside another app (e.g., VS Code, Terminal), the parent app appears in the list instead, not the executable itself. 2. Bundled App Version (from the reproduction project) I packaged the same code into a simple .app bundle (ScreenCaptureApp.app). When running the app: The same permission prompt appears. Pre-permission screenshots show the desktop background. Post-permission screenshots capture the full display. The app does appear under “Screen & System Audio Recording”. This bundle uses the same underlying executable — the only difference is packaging. Hypothesis macOS 26.1 (Tahoe) appears to require app bundles for an item to be shown in the Screen Recording privacy UI. Plain executables: still request and receive permission, still function correctly after permission is granted, but do not appear in the System Settings list. This may be an intentional change, undocumented behavior, or a regression. Reproduction Project The reproduction project includes: screen_capture.go A simple Go program that captures screenshots in a loop. screen_capture_executable Plain executable built from the Go source. ScreenCaptureApp.app/ App bundle containing the same executable. build.sh Builds both the plain executable and the app bundle. Permission reset and TCC testing scripts. The project demonstrates the behavior consistently. Steps to Reproduce Plain Executable Build: ./build.sh Reset screen capture permissions: sudo tccutil reset ScreenCapture Run: ./screen_capture_executable Before granting: screenshots show desktop only. Grant permission when prompted. After granting: full screenshots. Executable does not appear in “Screen & System Audio Recording”. Bundled App Build (if not already built): ./build.sh Reset permissions (optional): sudo tccutil reset ScreenCapture Run: open ScreenCaptureApp.app Before granting: screenshots show desktop. After granting: full screenshots. App bundle appears in the System Settings list. Additional Check I also tested launching the plain executable as a child process of another executable, similar to how some software architectures work. Result: Permission prompt appears Permission can be granted Executable still does not appear in the UI, even though TCC tracks it internally → consistent with the plain-executable behaviour. This reinforces that only app bundles are listed. Questions for Apple Is the removal of plain executables from “Screen & System Audio Recording” an intentional change in macOS Tahoe? If so, does Apple now require all screen-recording capable binaries to be packaged as .app bundles for the UI to display them? Is there a supported method for making a plain executable (launched by a parent process) appear in the list? If this is not intentional, what is the recommended path for reporting this as a regression? Files Unfortunately, I have discovered the zip file that contains my reproduction project can't be directly uploaded here. Here is a Google Drive link instead: https://drive.google.com/file/d/1sXsr3Q0g6_UzlOIL54P5wbS7yBkpMJ7A/view?usp=sharing Thank you for taking the time to review this. Any insight into whether this change is intentional or a regression would be very helpful.