In case when I have locked white balance and custom exposure, on black background when I introduce new object in view, both objects become brighter. How to turn off this feature or compensate for that change in a performant way? This is how I configure the session, note that Im setting a video format which supports at least 180 fps which is required for my needs. private func configureSession() { self.sessionQueue.async { [self] in //MARK: Init session guard let session = try? validSession() else { fatalError("Session is unexpectedly nil") } session.beginConfiguration() guard let device = AVCaptureDevice.default(AVCaptureDevice.DeviceType.builtInWideAngleCamera, for:AVMediaType.video, position: .back) else { fatalError("Video Device is unexpectedly nil") } guard let videoDeviceInput: AVCaptureDeviceInput = try? AVCaptureDeviceInput(device:device) else { fatalError("videoDeviceInput is unexpectedly nil") } guard session.canAddInput(videoDeviceInput) else { fatalError("videoDeviceInput could not be added") } session.addInput(videoDeviceInput) self.videoDeviceInput = videoDeviceInput self.videoDevice = device //MARK: Connect session IO let dataOutput = AVCaptureVideoDataOutput() dataOutput.setSampleBufferDelegate(self, queue: sampleBufferQueue) session.automaticallyConfiguresCaptureDeviceForWideColor = false guard session.canAddOutput(dataOutput) else { fatalError("Could not add video data output") } session.addOutput(dataOutput) dataOutput.alwaysDiscardsLateVideoFrames = true dataOutput.videoSettings = [ String(kCVPixelBufferPixelFormatTypeKey): pixelFormat.rawValue ] if let captureConnection = dataOutput.connection(with: .video) { captureConnection.preferredVideoStabilizationMode = .off captureConnection.isEnabled = true } else { fatalError("No Capture Connection for the session") } //MARK: Configure AVCaptureDevice do { try device.lockForConfiguration() } catch { fatalError(error.localizedDescription) } if let format = format(fps: fps, minWidth: minWidth, format: pixelFormat) { // 180FPS, YUV layout device.activeFormat = format device.activeVideoMinFrameDuration = CMTime(value: 1, timescale: CMTimeScale(fps)) device.activeVideoMaxFrameDuration = CMTime(value: 1, timescale: CMTimeScale(fps)) } else { fatalError("Compatible format not found") } device.activeColorSpace = .sRGB device.isGlobalToneMappingEnabled = false device.automaticallyAdjustsVideoHDREnabled = false device.automaticallyAdjustsFaceDrivenAutoExposureEnabled = false device.isFaceDrivenAutoExposureEnabled = false device.setFocusModeLocked(lensPosition: 0.4) device.isSubjectAreaChangeMonitoringEnabled = false device.exposureMode = AVCaptureDevice.ExposureMode.custom let exp = CMTime(value: Int64(40), timescale: 100_000) let isoValue = min(max(40, device.activeFormat.minISO), device.activeFormat.maxISO) device.setExposureModeCustom(duration: exp, iso: isoValue) { t in } device.setWhiteBalanceModeLocked(with: AVCaptureDevice.WhiteBalanceGains(redGain: 1.0, greenGain: 1.0, blueGain: 1.0)) { (timestamp:CMTime) -> Void in } device.unlockForConfiguration() session.commitConfiguration() onAVSessionReady() } } This post (https://stackoverflow.com/questions/34511431/ios-avfoundation-different-photo-brightness-with-the-same-manual-exposure-set) suggests that the effect can be mitigated by settings camera exposure to .locked right after setting device.setExposureModeCustom(). This works properly only if used with async api and still does not influence the effect. Async approach: private func onAVSessionReady() { guard let device = device() else { fatalError("Device is unexpectedly nil") } guard let sesh = try? validSession() else { fatalError("Device is unexpectedly nil") } MCamSession.shared.activeFormat = device.activeFormat MCamSession.shared.currentDevice = device self.observer = SPSDeviceKVO(device: device, session: sesh) self.start() Task { await lockCamera(device) } } private func lockCamera(_ device: AVCaptureDevice) async { do { try device.lockForConfiguration() } catch { fatalError(error.localizedDescription) } _ = await device.setFocusModeLocked(lensPosition: 0.4) let exp = CMTime(value: Int64(40), timescale: 100_000) let isoValue = min(max(40, device.activeFormat.minISO), device.activeFormat.maxISO) _ = await device.setExposureModeCustom(duration: exp, iso: isoValue) _ = await device.setWhiteBalanceModeLocked(with: AVCaptureDevice.WhiteBalanceGains(redGain: 1.0, greenGain: 1.0, blueGain: 1.0)) device.exposureMode = AVCaptureDevice.ExposureMode.locked device.unlockForConfiguration() } private func configureSession() { // same session init as before ... onAVSessionReady() }

As some have clocked, this was added in a recent SDK release... kCMVideoCodecType_AV1 ...does anyone know if and when AV1 decode support, even if software-only, is going to be available on Apple platforms? At the moment, one must decode using dav1d, (which is pretty performant, to be fair) but are we expecting at least software AV1 support on existing hardware any time soon, does anybody know?

Can you provide any information on why we might be getting OSStatus error code 268435465 while using Apple's Video Toolbox framework functionalities and how can we avoid getting it?

was curious if hardware accelerated encoding was only h264 supported?

0 VideoToolbox ___vtDecompressionSessionRemote_DecodeFrameCommon_block_invoke_2() 1 libdispatch.dylib __dispatch_client_callout() 2 libdispatch.dylib __dispatch_client_callout() 3 libdispatch.dylib __dispatch_lane_barrier_sync_invoke_and_complete() 4 VideoToolbox _vtDecompressionSessionRemote_DecodeFrameCommon() After the release of iOS 17 & iOS 16.7, thousands of crashes were added as videotoolbox each day. Has anyone encountered similar problems?

Hello, I'm trying to play an H.265 video stream, containing no I-frames but using GDR. No video is shown using VTDecompressionSessionDecodeFrame, other H.265 videos work fine. In the device log I can see the following errors: AppleAVD: VADecodeFrame(): isRandomAccessSkipPicture fail followed by An unknown error occurred (-17694) Is it possible to play such a video stream? Do I need to configure CMVideoFormatDescription in some way to enable it? Thanks for your help.

I use VTCompressionSession and set the average bit rate by kVTCompressionPropertyKey_AverageBitRate and kVTCompressionPropertyKey_DataRateLimits. The code like this: VTCompressionSessionRef vtSession = session; if (vtSession == NULL) { vtSession = _encoderSession; } if (vtSession == nil) { return; } int tmp = bitrate; int bytesTmp = tmp * 0.15; int durationTmp = 1; CFNumberRef bitrateRef = CFNumberCreate(NULL, kCFNumberSInt32Type, &tmp); CFNumberRef bytes = CFNumberCreate(NULL, kCFNumberSInt32Type, &bytesTmp); CFNumberRef duration = CFNumberCreate(NULL, kCFNumberSInt32Type, &durationTmp); if ([self isSupportPropertyWithSession:vtSession key:kVTCompressionPropertyKey_AverageBitRate]) { [self setSessionPropertyWithSession:vtSession key:kVTCompressionPropertyKey_AverageBitRate value:bitrateRef]; }else { NSLog(@"Video Encoder: set average bitRate error"); } NSLog(@"Video Encoder: set bitrate bytes = %d, _bitrate = %d",bytesTmp, bitrate); CFMutableArrayRef limit = CFArrayCreateMutable(NULL, 2, &kCFTypeArrayCallBacks); CFArrayAppendValue(limit, bytes); CFArrayAppendValue(limit, duration); if([self isSupportPropertyWithSession:vtSession key:kVTCompressionPropertyKey_DataRateLimits]) { OSStatus ret = VTSessionSetProperty(vtSession, kVTCompressionPropertyKey_DataRateLimits, limit); if(ret != noErr){ NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:ret userInfo:nil]; NSLog(@"Video Encoder: set DataRateLimits failed with %s",error.description.UTF8String); } }else { NSLog(@"Video Encoder: set data rate limits error"); } CFRelease(bitrateRef); CFRelease(limit); CFRelease(bytes); CFRelease(duration); } This work fine on iOS16 and below. But on iOS17 the bitrate of the generate video file is much lower than the value I set. For exmaple, I set biterate 600k but on iOS17 the encoded video bitrate is lower than 150k. What went wrong?

There is the crash on iOS17 beta7/beta8/RC/public. Crash Stack

Same as title

Use iOS17 VTCompressionSessionEncodeFrame beta5 version of video coding, if App switch back to the foreground from the background, Can appear kVTInvalidSessionErr reset according to the error invoked when VTCompressionSessionInvalidate cause gets stuck, if anyone else encountered this kind of situation, how should solve?

When I doing hardware videotoolbox encoding, it always failed in some mac machines. But it works in some other mac machines. I can not find the different of the machines. In problem machine, when I tried use ffmpeg executable do h265/h264 hardware encoding directly, it has issue. tried use ffmpeg executable h265/h264 software encoding, it has no issue. The error is like this: ./ffmpeg -f rawvideo -s 352x288 -pix_fmt yuv420p -i akiyo_cif.yuv -c:v hevc_videotoolbox test.mp4 [rawvideo @ 0x7f80b9704780] Estimating duration from bitrate, this may be inaccurate Input #0, rawvideo, from 'akiyo_cif.yuv': Duration: 00:00:12.00, start: 0.000000, bitrate: 30412 kb/s Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 352x288, 30412 kb/s, 25 tbr, 25 tbn Stream mapping: Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (hevc_videotoolbox)) Press [q] to stop, [?] for help [hevc_videotoolbox @ 0x7f80ba905640] Error encoding frame: -12905 [hevc_videotoolbox @ 0x7f80ba905640] popping: -542398533 [vost#0:0/hevc_videotoolbox @ 0x7f80ba9052c0] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height Conversion failed! Did more tests in problem machine and Check the processes in mac. The first launched VTEncoderXPCService process always has something wrong. The later launched VTEncoderXPCService can work as expected Hi Apple Expert, Do you know what's the reason is? Thanks.

To my surprise, there is nothing information about VTPixelRotationSession on the Internet. I tried Baidu and Bing, and I got nothing about it. Here is my code: var rotationSession: VTPixelRotationSession! override func viewDidLoad() { let state = VTPixelRotationSessionCreate(kCFAllocatorDefault, &rotationSession) if state != 0 { return } VTSessionSetProperty(rotationSession, key: kVTPixelRotationPropertyKey_Rotation, value: kVTRotation_CW90) } func rotate(sourceBuffer: CVImageBuffer) { let pixelFormat = CVPixelBufferGetPixelFormatType(sourceBuffer) var rotatedImage: CVPixelBuffer? CVPixelBufferCreate(kCFAllocatorDefault, 640, 480, pixelFormat, nil, &rotatedImage) let state = VTPixelRotationSessionRotateImage(rotationSession, sourceBuffer, rotatedImage) if state == kCVReturnSuccess { ...... } } The rotatedImageBuffer does not get any pixels of the sourceImageBuffer after calling VTPixelRotationSessionRotateImage. Anyone can explain this issue? Apple official does not give any examples about it.

Hi, I'm using VideoToolBox for h264 or HEVC video encoding. Thanks proper algorithm I used to change dynamically the video bitrate (by adjusting AverageBitrate and DataRateLimits) When running application on iOS 16.4 and further, video bitrate is no more respected by video encoder. Obtained bitrate is generally 70% percent lower than expected value. Is there a new setting to retrieve previous video encoder behaviour? Best regards

Hello, I am attempting to simultaneously stream video to a remote client and run inference on a neural network (on the same video frame) locally. I have done this in other platforms, using Gstreamer on linux and on android using libstreaming for compression and packetization. I've attempted this now on iPhone using ffmpeg to stream and a capture session to feed the neural network but I run into the problem of multiple camera access. Most of the posts I see are concerned with receiving RTP streams in iOS but I need to do the opposite. As I am new to iOS and Swift I was hoping someone could provide method for RTP packetization? Any library recommendations or example code for something similar? Best,

Hello, developers and expert, trying to implement encoding video simply using tencent sdk but I got error, and continuation doesn't return any progress or result. anyone help me please... what i am missing? SWIFT TASK CONTINUATION MISUSE: generateVideo(_:) leaked its continuation! 2023-03-30 12:07:15.694914+0900 Metacellar (dev)[87044:19556775] SWIFT TASK CONTINUATION MISUSE: generateVideo(_:) leaked its continuation! func encodeVideo() async throws { if let video { let encoded = try await TencentVideoEncoder.generateVideo(video) } } import TXLiteAVSDK_Professional class TencentVideoEncoder: NSObject { static func generateVideo(_ original: AVAsset) async throws -> AVAsset { let inputPath = try await Cache.saveToLocal( asset: original, cache: Cache.temporary, cacheKey: UUID().uuidString) let outputPath = MetaCellarCache.temporary.cachePath(forKey: UUID().uuidString)! let editor = TXVideoEditer() editor.setVideoBitrate(2300) editor.setVideoPath(inputPath) editor.setCutFromTime(0, toTime: 10) return try await withCheckedThrowingContinuation { continuation in let listener = Listener(onComplete: { result in switch result.retCode { case .GENERATE_RESULT_OK: continuation.resume(returning: AVAsset(url: URL(fileURLWithPath: outputPath))) default: continuation.resume(throwing: NSError(domain: "Encoding Failed", code: result.retCode.rawValue)) } }, onProgress: { progress in print(progress) }) editor.generateDelegate = listener editor.generateVideo(withTwoPass: .VIDEO_COMPRESSED_720P, videoOutputPath: outputPath) } } class Listener: NSObject, TXVideoGenerateListener { private var onComplete: (TXGenerateResult) -> () private var onProgress: (Float) -> () init(onComplete: @escaping (TXGenerateResult) -> (), onProgress: @escaping (Float) -> ()) { self.onComplete = onComplete self.onProgress = onProgress } func onGenerateComplete(_ result: TXGenerateResult) { onComplete(result) } func onGenerateProgress(_ progress: Float) { onProgress(progress) } } }

I am trying to run the videotoolbox api. It works fine on both emulator (tested Iphone 14 Pro and Iphone SE) and osx Macbook. When running on the Iphone the synchronous method VTCompressionSessionCreate does not return but gets stuck. This is hitting pause in XCode after a few seconds of wait time: The code leading up to this: " let imageBufferAttributes = [         kCVPixelBufferWidthKey: NSNumber(value: width),         kCVPixelBufferHeightKey: NSNumber(value: height),         kCVPixelBufferPixelFormatTypeKey: NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)       ]               let encoderSpecification = [         kVTVideoEncoderSpecification_EnableLowLatencyRateControl: NSNumber(value: true),       ]               let compressionSessionOut = UnsafeMutablePointer<VTCompressionSession?>.allocate(capacity: 1)               // Callback       let encodeCallback: VTCompressionOutputCallback = { outputCallbackRefCon, sourceFrameRefCon, status, infoFlags, sampleBuffer in         if CMSampleBufferDataIsReady(sampleBuffer!) {           let encoder: Encoder = Unmanaged<Encoder>.fromOpaque(sourceFrameRefCon!).takeUnretainedValue()           encoder.writeData(sampleBuffer: sampleBuffer!, infoFlags: infoFlags)         }       }       // Create session       var status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault,                           width: Int32(width),                           height: Int32(height),                           codecType: codecType, //kCMVideoCodecType_H264,                           encoderSpecification: nil,//encoderSpecification as CFDictionary?,                           imageBufferAttributes: imageBufferAttributes as CFDictionary?,                           compressedDataAllocator: kCFAllocatorDefault,                           outputCallback: encodeCallback,                           refcon: Unmanaged.passUnretained(self).toOpaque(),                           compressionSessionOut: compressionSessionOut) " I have searched the web for anyone else seeing this but so far not found anyone. Is my hardware broken? It is an old device (I am waiting for a newer device).

Hey folks - I'm a bit clueless about this specific topic, so bear with me please. I make use of the recording/streaming app called OBS Studio on both my Windows and Mac rigs. OBS supports hardware encoding to h.264 for recording and streaming so that you don't have to lean on its internal FFMpeg calls to transcode for you. Big CPU saver there. I note when I'm attempting to stream to YouTube, for instance, that YT reports back that I'm sending all sorts of wonky bitrates. OBS lets me set a "bitrate" and "bitrate limit", which I have to 50Mbit/sec. YouTube will randomly report back that I'm sending it 70Mbit/sec. Or 90+ Mbit/sec. This is brief, but it happens repeatedly. Challenge: streaming services hate VBR. And according to the OBS devs I've talked to, they seem to think that Apple's VT encoders only support VBR. Is that true? Is there any way to force CBR so that my Mac only ever sends 50Mbit/sec, no more, no less? Thanks.

Video conferencing software, previous version sound normal. After iOS16, the feedback session was echoing. audioUnit KAudioUnitSubType_VoiceProcessingIO has been set. I understand that iOS hardware echo cancellation only needs to set kAudioUnitSubType_VoiceProcessingIO. Why does it echo after iOS16 comes out?

The sources that I have seen for VideoDecompressionSession use H.264 as the example decoder. I suspect that H.265 could also be decoded with VideoDecompressionSession. Can anyone confirm that?

When i'm trying ‘MobileVLCKit’ to load the rtsp stream (remote camera), iPhone asks to grant access for local network. If Allow access, then the rtsp camera stream don’t play and if Don’t  Allow access, then the rtsp camera stream working fine, and also working on simulator