I am using the official website API to decode now, the callback did not trigger.

我使用VideoToolBox库进行编码后再通过NDI发送到网络上，是可以成功再苹果电脑上接收到ndi源屏显示画面的，但是在windows上只能ndi源名称，并没有画面显示。 我想知道是不是使用VideoToolBox库无法在windows上进行正确编码，这个问题需要如何解决

Who can I contact that can remove the gray frame from around the full screen video player on my iOS mobile application? This is an Apple iOS feature that I have no control over. The screenshot attached below shows the full screen view of a video when the iOS mobile phone is held sideways. The issue is the big gray frame that is around the video, is taking up too much space from the video and it needs to be removed so the video can be fully screened.

Is there a way to play a specific rectangular region of interest of a video in an arbitrarily-sized view? Let's say I have a 1080p video but I'm only interested in a sub-region of the full frame. Is there a way to specify a source rect to be displayed in an arbitrary view (SwiftUI view, ideally), and have it play that in real time, without having to pre-render the cropped region? Update: I may have found a solution here: img DOT ly/blog/trim-and-crop-video-in-swift/ (Apple won't allow that URL for some dumb reason)

Hi, I've started learning swiftUI a few months ago, and now I'm trying to build my first app :) I am trying to display VTT subtitles from an external URL into a streaming video using AVPlayer and AVMutableComposition. I have been trying for a few days, checking online and on Apple's documentation, but I can't manage to make it work. So far, I managed to display the subtitles, but there is no video or audio playing... Could someone help? Thanks in advance, I hope the code is not too confusing. // EpisodeDetailView.swift // OroroPlayer_v1 // // Created by Juan Valenzuela on 2023-11-25. // import AVKit import SwiftUI struct EpisodeDetailView4: View { @State private var episodeDetailVM = EpisodeDetailViewModel() let episodeID: Int @State private var player = AVPlayer() @State private var subs = AVPlayer() var body: some View { VideoPlayer(player: player) .ignoresSafeArea() .task { do { try await episodeDetailVM.fetchEpisode(id: episodeID) let episode = episodeDetailVM.episodeDetail guard let videoURLString = episode.url else { print("Invalid videoURL or missing data") return } guard let subtitleURLString = episode.subtitles?[0].url else { print("Invalid subtitleURLs or missing data") return } let videoURL = URL(string: videoURLString)! let subtitleURL = URL(string: subtitleURLString)! let videoAsset = AVURLAsset(url: videoURL) let subtitleAsset = AVURLAsset(url: subtitleURL) let movieWithSubs = AVMutableComposition() let videoTrack = movieWithSubs.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid) let audioTrack = movieWithSubs.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid) let subtitleTrack = movieWithSubs.addMutableTrack(withMediaType: .text, preferredTrackID: kCMPersistentTrackID_Invalid) // if let videoTrackItem = try await videoAsset.loadTracks(withMediaType: .video).first { try await videoTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)), of: videoTrackItem, at: .zero) } if let audioTrackItem = try await videoAsset.loadTracks(withMediaType: .audio).first { try await audioTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)), of: audioTrackItem, at: .zero) } if let subtitleTrackItem = try await subtitleAsset.loadTracks(withMediaType: .text).first { try await subtitleTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)), of: subtitleTrackItem, at: .zero) } let playerItem = AVPlayerItem(asset: movieWithSubs) player = AVPlayer(playerItem: playerItem) let playerController = AVPlayerViewController() playerController.player = player playerController.player?.play() // player.play() } catch { print("Error: \(error.localizedDescription)") } } } } #Preview { EpisodeDetailView4(episodeID: 39288) }

When I upload a preview video to ASC (which conforms to preview specifications), the uploaded video it uploaded correctly in the first place. During upload to ASC it shows a blurred image of the first video frame. So far so good. But, once upload is finished the video turns into a "cloud" image and says it is currently processed. The problem is that it gets stuck in the status „currently processed“ forever. I waited a few days but processing did never end. To make it worse the landscape "cloud" image turns into a portrait when I come back to the ASC media center. The problem : is reproducible occurs on different video files occurs on different appIDs occurs on all iPhone resolutions This is a serious bug. I can’t finalise my app submission. Any ideas ?

Hello! I'm trying to display AVPlayerViewController in a separate WindowGroup - my main window opens a new window where the only element is a struct that implements UIViewControllerRepresentable for AVPlayerViewController: @MainActor public struct AVPlayerView: UIViewControllerRepresentable { public let assetName: String public init(assetName: String) { self.assetName = assetName } public func makeUIViewController(context: Context) -> AVPlayerViewController { let controller = AVPlayerViewController() controller.player = AVPlayer() return controller } public func updateUIViewController(_ controller: AVPlayerViewController, context: Context) { Task { if context.coordinator.assetName != assetName { let url = Bundle.main.url(forResource: assetName, withExtension: ".mp4") guard let url else { return } controller.player?.replaceCurrentItem(with: AVPlayerItem(url: url)) controller.player?.play() context.coordinator.assetName = assetName } } } public static func dismantleUIViewController(_ controller: AVPlayerViewController, coordinator: Coordinator) { controller.player?.pause() controller.player = nil } public func makeCoordinator() -> Coordinator { return Coordinator() } public class Coordinator: NSObject { public var assetName: String? } } WindowGroup(id: Window.videoPlayer.rawValue) { AVPlayerView(assetName: "wwdc") .onDisappear { print("DISAPPEAR") } } This displays the video player in non-inline mode and plays the video. The problem appears when I try to close the video player's window using the close button. Sound from the video continues playing in the background. I've tried to clean the state myself by using dismantleUIViewController and onDisapear methods, but they are not called by the system (it works correctly if a window doesn't contain AVPlayerView). This appear on Xcode 15.1 Beta 3 (I haven't tested it on other versions). Is there something I do incorrectly that is causing this issue, or is it a bug and I need to wait until its fixed?

We are currently working on a real-time, low-latency solution for video conferencing scenarios and have encountered some issues with the current implementation of the encoder. We need a feature enhancement for the Videotoolbox encoder. In our use case, we need to control the encoding quality, which requires setting the maximum encoding QP. However, the kVTCompressionPropertyKey_MaxAllowedFrameQP only takes effect in the kVTVideoEncoderSpecification_EnableLowLatencyRateControl mode. In this mode, when the maximum QP is limited and the bitrate is insufficient, the encoder will drop frames. Our desired scenario is for the encoder to not actively drop frames when the maximum QP is limited. Instead, when the bitrate is insufficient, the encoder should be able to encode the frame with the maximum QP, allowing the frame size to be larger. This would provide a more seamless experience for users in video conferencing situations, where maintaining consistent video quality is crucial. It is worth noting that Android has already implemented this feature in Android 12, which demonstrates the value and feasibility of this enhancement. We kindly request that you consider adding support for external control of frame dropping in the Videotoolbox encoder to accommodate our needs. This enhancement would greatly benefit our project and others that require real-time, low-latency video encoding solutions.

In MacOS14 system, the transparency of CMSampleBuffer is 0. Why after sending through the send function of CMIOExtensionStream, the receiving end uses AVCaptureSession to addOutput: AVCaptureVideoDataOutput object, AVCaptureVideoDataOutput sets videoSettings to kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA), the receiving end's receiving delegate method - (void)captureOutput:( AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection The transparency of the sampleBuffer returned is actually 255. The correct sampleBuffer transparency of the MacOS13 system is 0. In this case, how can I return the same as the 13 system? What needs to be set? Attribute?

I'm working on a MV-HEVC transcoder, based on the VTEncoderForTranscoding sample code. In swift the following code snippet generates a linker error on macOS 14.0 and 14.1. let err = VTCompressionSessionEncodeMultiImageFrame(compressionSession, taggedBuffers: taggedBuffers, presentationTimeStamp: pts, duration: .invalid, frameProperties: nil, infoFlagsOut: nil) { (status: OSStatus, infoFlags: VTEncodeInfoFlags, sbuf: CMSampleBuffer?) -> Void in outputHandler(status, infoFlags, sbuf, thisFrameNumber) } Error: ld: Undefined symbols: VideoToolbox.VTCompressionSessionEncodeMultiImageFrame(_: __C.VTCompressionSessionRef, taggedBuffers: [CoreMedia.CMTaggedBuffer], presentationTimeStamp: __C.CMTime, duration: __C.CMTime, frameProperties: __C.CFDictionaryRef?, infoFlagsOut: Swift.UnsafeMutablePointer<__C.VTEncodeInfoFlags>?, outputHandler: (Swift.Int32, __C.VTEncodeInfoFlags, __C.CMSampleBufferRef?) -> ()) -> Swift.Int32, referenced from: (3) suspend resume partial function for VTEncoderForTranscoding_Swift.(compressFrames in _FE7277D5F28D8DABDFC10EA0164D825D)(from: VTEncoderForTranscoding_Swift.VideoSource, options: VTEncoderForTranscoding_Swift.Options, expectedFrameRate: Swift.Float, outputHandler: @Sendable (Swift.Int32, __C.VTEncodeInfoFlags, __C.CMSampleBufferRef?, Swift.Int) -> ()) async throws -> () in VTEncoderForTranscoding.o Using VTCompressionSessionEncodeMultiImageFrameWithOutputHandler in ObjC doesn't trigger a linker error. Anybody knows how to get it to work in Swift?

Can you provide any information on why we might be getting OSStatus error code 268435465 while using Apple's Video Toolbox framework functionalities and how can we avoid getting it?

Recently I've been trying to play some AV1-encoded streams on my iPhone 15 Pro Max. First, I check for hardware support: VTIsHardwareDecodeSupported(kCMVideoCodecType_AV1); // YES Then I need to create a CMFormatDescription in order to pass it into a VTDecompressionSession. I've tried the following: { mediaType:'vide' mediaSubType:'av01' mediaSpecific: { codecType: 'av01' dimensions: 394 x 852 } extensions: {{ CVFieldCount = 1; CVImageBufferChromaLocationBottomField = Left; CVImageBufferChromaLocationTopField = Left; CVPixelAspectRatio = { HorizontalSpacing = 1; VerticalSpacing = 1; }; FullRangeVideo = 0; }} } but VTDecompressionSessionCreate gives me error -8971 (codecExtensionNotFoundErr, I assume). So it has something to do with the extensions dictionary? I can't find anywhere which set of extensions is necessary for it to work 😿. VideoToolbox has convenient functions for creating descriptions of AVC and HEVC streams (CMVideoFormatDescriptionCreateFromH264ParameterSets and CMVideoFormatDescriptionCreateFromHEVCParameterSets), but not for AV1. As of today I am using XCode 15.0 with iOS 17.0.0 SDK.

was curious if hardware accelerated encoding was only h264 supported?

0 VideoToolbox ___vtDecompressionSessionRemote_DecodeFrameCommon_block_invoke_2() 1 libdispatch.dylib __dispatch_client_callout() 2 libdispatch.dylib __dispatch_client_callout() 3 libdispatch.dylib __dispatch_lane_barrier_sync_invoke_and_complete() 4 VideoToolbox _vtDecompressionSessionRemote_DecodeFrameCommon() After the release of iOS 17 & iOS 16.7, thousands of crashes were added as videotoolbox each day. Has anyone encountered similar problems?

I use VTCompressionSession and set the average bit rate by kVTCompressionPropertyKey_AverageBitRate and kVTCompressionPropertyKey_DataRateLimits. The code like this: VTCompressionSessionRef vtSession = session; if (vtSession == NULL) { vtSession = _encoderSession; } if (vtSession == nil) { return; } int tmp = bitrate; int bytesTmp = tmp * 0.15; int durationTmp = 1; CFNumberRef bitrateRef = CFNumberCreate(NULL, kCFNumberSInt32Type, &tmp); CFNumberRef bytes = CFNumberCreate(NULL, kCFNumberSInt32Type, &bytesTmp); CFNumberRef duration = CFNumberCreate(NULL, kCFNumberSInt32Type, &durationTmp); if ([self isSupportPropertyWithSession:vtSession key:kVTCompressionPropertyKey_AverageBitRate]) { [self setSessionPropertyWithSession:vtSession key:kVTCompressionPropertyKey_AverageBitRate value:bitrateRef]; }else { NSLog(@"Video Encoder: set average bitRate error"); } NSLog(@"Video Encoder: set bitrate bytes = %d, _bitrate = %d",bytesTmp, bitrate); CFMutableArrayRef limit = CFArrayCreateMutable(NULL, 2, &kCFTypeArrayCallBacks); CFArrayAppendValue(limit, bytes); CFArrayAppendValue(limit, duration); if([self isSupportPropertyWithSession:vtSession key:kVTCompressionPropertyKey_DataRateLimits]) { OSStatus ret = VTSessionSetProperty(vtSession, kVTCompressionPropertyKey_DataRateLimits, limit); if(ret != noErr){ NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:ret userInfo:nil]; NSLog(@"Video Encoder: set DataRateLimits failed with %s",error.description.UTF8String); } }else { NSLog(@"Video Encoder: set data rate limits error"); } CFRelease(bitrateRef); CFRelease(limit); CFRelease(bytes); CFRelease(duration); } This work fine on iOS16 and below. But on iOS17 the bitrate of the generate video file is much lower than the value I set. For exmaple, I set biterate 600k but on iOS17 the encoded video bitrate is lower than 150k. What went wrong?

I want to show the user actual start and end dates of the video played on the AVPlayer time slider, instead of the video duration data. I would like to show something like this: 09:00:00 ... 12:00:00 (which indicates that the video started at 09:00:00 CET and ended at 12:00:00 CET), instead of: 00:00:00 ... 02:59:59. I would appreciate any pointers to this direction.

There is the crash on iOS17 beta7/beta8/RC/public. Crash Stack

Same as title

In case when I have locked white balance and custom exposure, on black background when I introduce new object in view, both objects become brighter. How to turn off this feature or compensate for that change in a performant way? This is how I configure the session, note that Im setting a video format which supports at least 180 fps which is required for my needs. private func configureSession() { self.sessionQueue.async { [self] in //MARK: Init session guard let session = try? validSession() else { fatalError("Session is unexpectedly nil") } session.beginConfiguration() guard let device = AVCaptureDevice.default(AVCaptureDevice.DeviceType.builtInWideAngleCamera, for:AVMediaType.video, position: .back) else { fatalError("Video Device is unexpectedly nil") } guard let videoDeviceInput: AVCaptureDeviceInput = try? AVCaptureDeviceInput(device:device) else { fatalError("videoDeviceInput is unexpectedly nil") } guard session.canAddInput(videoDeviceInput) else { fatalError("videoDeviceInput could not be added") } session.addInput(videoDeviceInput) self.videoDeviceInput = videoDeviceInput self.videoDevice = device //MARK: Connect session IO let dataOutput = AVCaptureVideoDataOutput() dataOutput.setSampleBufferDelegate(self, queue: sampleBufferQueue) session.automaticallyConfiguresCaptureDeviceForWideColor = false guard session.canAddOutput(dataOutput) else { fatalError("Could not add video data output") } session.addOutput(dataOutput) dataOutput.alwaysDiscardsLateVideoFrames = true dataOutput.videoSettings = [ String(kCVPixelBufferPixelFormatTypeKey): pixelFormat.rawValue ] if let captureConnection = dataOutput.connection(with: .video) { captureConnection.preferredVideoStabilizationMode = .off captureConnection.isEnabled = true } else { fatalError("No Capture Connection for the session") } //MARK: Configure AVCaptureDevice do { try device.lockForConfiguration() } catch { fatalError(error.localizedDescription) } if let format = format(fps: fps, minWidth: minWidth, format: pixelFormat) { // 180FPS, YUV layout device.activeFormat = format device.activeVideoMinFrameDuration = CMTime(value: 1, timescale: CMTimeScale(fps)) device.activeVideoMaxFrameDuration = CMTime(value: 1, timescale: CMTimeScale(fps)) } else { fatalError("Compatible format not found") } device.activeColorSpace = .sRGB device.isGlobalToneMappingEnabled = false device.automaticallyAdjustsVideoHDREnabled = false device.automaticallyAdjustsFaceDrivenAutoExposureEnabled = false device.isFaceDrivenAutoExposureEnabled = false device.setFocusModeLocked(lensPosition: 0.4) device.isSubjectAreaChangeMonitoringEnabled = false device.exposureMode = AVCaptureDevice.ExposureMode.custom let exp = CMTime(value: Int64(40), timescale: 100_000) let isoValue = min(max(40, device.activeFormat.minISO), device.activeFormat.maxISO) device.setExposureModeCustom(duration: exp, iso: isoValue) { t in } device.setWhiteBalanceModeLocked(with: AVCaptureDevice.WhiteBalanceGains(redGain: 1.0, greenGain: 1.0, blueGain: 1.0)) { (timestamp:CMTime) -> Void in } device.unlockForConfiguration() session.commitConfiguration() onAVSessionReady() } } This post (https://stackoverflow.com/questions/34511431/ios-avfoundation-different-photo-brightness-with-the-same-manual-exposure-set) suggests that the effect can be mitigated by settings camera exposure to .locked right after setting device.setExposureModeCustom(). This works properly only if used with async api and still does not influence the effect. Async approach: private func onAVSessionReady() { guard let device = device() else { fatalError("Device is unexpectedly nil") } guard let sesh = try? validSession() else { fatalError("Device is unexpectedly nil") } MCamSession.shared.activeFormat = device.activeFormat MCamSession.shared.currentDevice = device self.observer = SPSDeviceKVO(device: device, session: sesh) self.start() Task { await lockCamera(device) } } private func lockCamera(_ device: AVCaptureDevice) async { do { try device.lockForConfiguration() } catch { fatalError(error.localizedDescription) } _ = await device.setFocusModeLocked(lensPosition: 0.4) let exp = CMTime(value: Int64(40), timescale: 100_000) let isoValue = min(max(40, device.activeFormat.minISO), device.activeFormat.maxISO) _ = await device.setExposureModeCustom(duration: exp, iso: isoValue) _ = await device.setWhiteBalanceModeLocked(with: AVCaptureDevice.WhiteBalanceGains(redGain: 1.0, greenGain: 1.0, blueGain: 1.0)) device.exposureMode = AVCaptureDevice.ExposureMode.locked device.unlockForConfiguration() } private func configureSession() { // same session init as before ... onAVSessionReady() }

I have multiple AVAssets with that I am trying to merge together into a single video track using AVComposition. What I'm doing is iterating over my avassets and inserting them in to a single AVCompositionTrack like so: - (AVAsset *)combineAssets { // Create a mutable composition AVMutableComposition *composition = [AVMutableComposition composition]; AVMutableCompositionTrack *compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid]; AVMutableCompositionTrack *compositionAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; // Keep track of time offset CMTime currentOffset = kCMTimeZero; for (AVAsset *audioAsset in _audioSegments) { AVAssetTrack *audioTrack = [[audioAsset tracksWithMediaType:AVMediaTypeAudio] firstObject]; // Add the audio track to the composition audio track NSError *audioError; [compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, audioAsset.duration) ofTrack:audioTrack atTime:currentOffset error:&audioError]; if (audioError) { NSLog(@"Error combining audio track: %@", audioError.localizedDescription); return nil; } currentOffset = CMTimeAdd(currentOffset, audioAsset.duration); } // Reset offset to do the same with videos. currentOffset = kCMTimeZero; for (AVAsset *videoAsset in _videoSegments) { { // Get the video track AVAssetTrack *videoTrack = [[videoAsset tracksWithMediaType:AVMediaTypeVideo] firstObject]; // Add the video track to the composition video track NSError *videoError; [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoTrack.timeRange.duration) ofTrack:videoTrack atTime:currentOffset error:&videoError]; if (videoError) { NSLog(@"Error combining audio track: %@", videoError.localizedDescription); return nil; } // Increment current offset by asset duration currentOffset = CMTimeAdd(currentOffset, videoTrack.timeRange.duration); } } return composition; } The issue is that when I export the composition using an AVExportSession I notice that there's a black frame between the merged segments in the track. In other words, if there were two 30second AVAssets merged into the composition track to create a 60 second video. You would see a black frame for a split second at the 30 second mark where the two assets combine. I don't really want to re encode the assets I just want to stitch them together. How can I fix the black frame issue?.