User Profile

Issue I'm using AVFoundation to implement a Camera that is able to record videos while running special AI processing. Having an AVCaptureMovieFileOutput (for video recording) and a AVCaptureVideoDataOutput (for processing AI) running at the same time is not supported (see https://stackoverflow.com/q/4944083/5281431), so I have decided to use a single AVCaptureVideoDataOutput which is able to record videos to a file while running the AI processing in the same captureOutput(...) callback. To my surprise, doing that drastically increases RAM usage from 58 MB to 187 MB (!!!), and CPU from 3-5% to 7-12% while idle. While actually recording, the RAM goes up even more (260 MB!). I am wondering what I did wrong here, since I disabled all the AI processing and just compared the differences between AVCaptureMovieFileOutput and AVCaptureVideoDataOutput. My code: AVCaptureMovieFileOutput Setup swift if let movieOutput = self.movieOutput { captureSession.removeOutput(movieOutput) } movieOutput = AVCaptureMovieFileOutput() captureSession.addOutput(movieOutput!) Delegate (well there is none, AVCaptureMovieFileOutput handles all that internally) Benchmark When idle, so not recording at all: RAM: 56 MB CPU: 3-5% When recording using AVCaptureMovieFileOutput.startRecording: RAM: 56 MB (how???) CPU: 20-30% AVCaptureVideoDataOutput Setup swift // Video if let videoOutput = self.videoOutput { captureSession.removeOutput(videoOutput) self.videoOutput = nil } videoOutput = AVCaptureVideoDataOutput() videoOutput!.setSampleBufferDelegate(self, queue: videoQueue) videoOutput!.alwaysDiscardsLateVideoFrames = true captureSession.addOutput(videoOutput!) // Audio if let audioOutput = self.audioOutput { captureSession.removeOutput(audioOutput) self.audioOutput = nil } audioOutput = AVCaptureAudioDataOutput() audioOutput!.setSampleBufferDelegate(self, queue: audioQueue) captureSession.addOutput(audioOutput!) Delegate swift extension CameraView: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioDataOutputSampleBufferDelegate { public final func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from _: AVCaptureConnection) { // empty } public final func captureOutput(_ captureOutput: AVCaptureOutput, didDrop buffer: CMSampleBuffer, from _: AVCaptureConnection) { // empty } } yes, they are literally empty methods. My RAM and CPU usage is still that high without doing any work here. Benchmark When idle, so not recording at all: RAM: 151-187 MB CPU: 7-12% When recording using a custom AVAssetWriter: RAM: 260 MB CPU: 64% Why is the AVCaptureMovieFileOutput so much more efficient than an empty AVCaptureVideoDataOutput? Also, why does it's RAM not go up at all when recording, compared to how my AVAssetWriter implementation alone consumes 80 MB? Here's my custom AVAssetWriter implementation: [RecordingSession.swift](https://github.com/cuvent/react-native-vision-camera/blob/frame-processors/ios/RecordingSession.swift), and here's where I call it - https://github.com/cuvent/react-native-vision-camera/blob/a48ca839e93e6199ad731f348e19427774c92821/ios/CameraView%2BRecordVideo.swift#L16-L86. Any help appreciated!

Hi! I have created a Camera using AVFoundation which is able to record video and audio using AVCaptureVideoDataOutput and AVCaptureAudioDataOutput. I create my capture session, attach all inputs and the video- and audio-data outputs, and the Camera then sits idle. The user is now able to start a video recording. Problem The problem with this is that immediately after I start the capture session, the background music stutters. This is really annoying, since the Camera is the start-screen in our app and I want to delay the stuttering until the Audio is actually needed, which is when the user starts recording a video. I know that this is somehow possible because Snapchat works that way - you open the App and background audio smoothly continues to play. Once you start recording, there is a small stutter on the background music, but the Camera smoothly operates and starts recording once the short stutter is over. My code: func configureSession() { captureSession.beginConfiguration() // Video, Photo and Audio Inputs ... // Video Output ... // Audio Output audioOutput = AVCaptureAudioDataOutput() guard captureSession.canAddOutput(audioOutput!) else { throw CameraError.parameter(.unsupportedOutput(outputDescriptor: "audio-output")) } audioOutput!.setSampleBufferDelegate(self, queue: audioQueue) captureSession.addOutput(audioOutput!) try AVAudioSession.sharedInstance().setCategory(AVAudioSession.Category.playAndRecord, options: [.mixWithOthers, .allowBluetoothA2DP, .defaultToSpeaker, .allowAirPlay]) captureSession.commitConfiguration() } What I tried Delay configuring the AVAudioSession.sharedInstance() I tried to first configure the AVAudioSession.sharedInstance with the category AVAudioSession.Category.playback, and switch to .playAndRecord once I want to start recording audio. This didn't work and the AVCaptureSessionRuntimeError event gets invoked with the Error code -10851, which means kAudioUnitErr_InvalidPropertyValue. I think this means that the AVCaptureAudioDataOutput is not allowed to record from the Audio Session, but I don't event want to do that right now - it should just be idle. Delay adding the AVCaptureAudioDataOutput output I tried to not add the audio output (AVCaptureAudioDataOutput) in the beginning, and only add it "on-demand" once the user starts recording, and while that worked fine for the background music (no stutter when starting, only short stutter once the user starts recording, exactly how I want it), it made the Preview freeze for a short amount of time (because the Capture Session is being reconfigured via beginConfiguration + audio output adding + commitConfiguration) Does anyone know how it's possible to achieve what I'm trying to do here - or how Snapchat does it? Any help appreciated, thanks!

I'm trying to create a camera capture session that has the following features: Preview Photo capture Video capture Realtime frame processing (AI) While the first two things are not a problem, I haven't found a way to make the last two separately. Currently, I use a single AVCaptureVideoDataOutput and run the video recording first, then the frame processing in the same function, in the same queue. (see code here - https://github.com/cuvent/react-native-vision-camera/blob/49ae9844da0daf8ce259c2b2f482e1baed4a82c8/ios/CameraView%2BAVCaptureSession.swift#L16-L134 and here - https://github.com/cuvent/react-native-vision-camera/blob/49ae9844da0daf8ce259c2b2f482e1baed4a82c8/ios/CameraView%2BRecordVideo.swift#L121-L146) The only problem with this is that the video capture captures 4k video, and I don't really want the frame processor to receive 4k buffers as that is going to be very slow and blocks the video recording (frame drops). Ideally I want to create one AVCaptureDataOutput for 4k video recording, and another one that receives frames in a lower (preview?) resolution - but you cannot use two AVCaptureDataOutputs in the same capture session. I thought maybe I could "hook into" the Preview layer to receive the CMSampleBuffers from there, just like the captureOutput(...) func, since those are in preview-sized resolutions, does anyone know if that is somehow possible?