VideoToolbox

RSS for tag

Work directly with hardware-accelerated video encoding and decoding capabilities using VideoToolbox.

Posts under VideoToolbox tag

117 Posts

Post

Replies

Boosts

Views

Activity

VTCompressionSessionCreate hangs on Iphone 8
I am trying to run the videotoolbox api. It works fine on both emulator (tested Iphone 14 Pro and Iphone SE) and osx Macbook. When running on the Iphone the synchronous method VTCompressionSessionCreate does not return but gets stuck. This is hitting pause in XCode after a few seconds of wait time: The code leading up to this: " let imageBufferAttributes = [         kCVPixelBufferWidthKey: NSNumber(value: width),         kCVPixelBufferHeightKey: NSNumber(value: height),         kCVPixelBufferPixelFormatTypeKey: NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)       ]               let encoderSpecification = [         kVTVideoEncoderSpecification_EnableLowLatencyRateControl: NSNumber(value: true),       ]               let compressionSessionOut = UnsafeMutablePointer<VTCompressionSession?>.allocate(capacity: 1)               // Callback       let encodeCallback: VTCompressionOutputCallback = { outputCallbackRefCon, sourceFrameRefCon, status, infoFlags, sampleBuffer in         if CMSampleBufferDataIsReady(sampleBuffer!) {           let encoder: Encoder = Unmanaged<Encoder>.fromOpaque(sourceFrameRefCon!).takeUnretainedValue()           encoder.writeData(sampleBuffer: sampleBuffer!, infoFlags: infoFlags)         }       }       // Create session       var status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault,                           width: Int32(width),                           height: Int32(height),                           codecType: codecType, //kCMVideoCodecType_H264,                           encoderSpecification: nil,//encoderSpecification as CFDictionary?,                           imageBufferAttributes: imageBufferAttributes as CFDictionary?,                           compressedDataAllocator: kCFAllocatorDefault,                           outputCallback: encodeCallback,                           refcon: Unmanaged.passUnretained(self).toOpaque(),                           compressionSessionOut: compressionSessionOut) " I have searched the web for anyone else seeing this but so far not found anyone. Is my hardware broken? It is an old device (I am waiting for a newer device).
5
1
2.3k
Mar ’23
VideoToolbox h.264 Encoder: CBR Possible?
Hey folks - I'm a bit clueless about this specific topic, so bear with me please. I make use of the recording/streaming app called OBS Studio on both my Windows and Mac rigs. OBS supports hardware encoding to h.264 for recording and streaming so that you don't have to lean on its internal FFMpeg calls to transcode for you. Big CPU saver there. I note when I'm attempting to stream to YouTube, for instance, that YT reports back that I'm sending all sorts of wonky bitrates. OBS lets me set a "bitrate" and "bitrate limit", which I have to 50Mbit/sec. YouTube will randomly report back that I'm sending it 70Mbit/sec. Or 90+ Mbit/sec. This is brief, but it happens repeatedly. Challenge: streaming services hate VBR. And according to the OBS devs I've talked to, they seem to think that Apple's VT encoders only support VBR. Is that true? Is there any way to force CBR so that my Mac only ever sends 50Mbit/sec, no more, no less? Thanks.
2
0
4.7k
Mar ’23
h264 encoder with wrong frame order
I use VideoToolbox HW h.264 encoder with M1 MacBook for screen mirroring. I need run encoder with minimal delay mode. I use these values as encoderSpecification kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder:true kVTCompressionPropertyKey_ProfileLevel:kVTProfileLevel_H264_Baseline_AutoLevel kVTCompressionPropertyKey_RealTime:true kVTCompressionPropertyKey_AllowFrameReordering:false I set presentation timestamp with current time. In compressed callback, I got encoded frame with wrong order. [13.930533]encode pts=...13.930511 [13.997633]encode pts=...13.997617 [14.013678]compress callback with pts=...13.997617 dts=...13.930511 [14.023443]compress callback with pts=...13.930511 dts=...13.997617 in[]:log time, pts:presentation timestamp, dts:decode timestamp AllowFrameReordering is not working as I expected. If I need to set other property, please let me know. I also does not like buffering 2 video frames. If you know settings for no buffering frame, please let me know.
2
0
1.9k
Jan ’23
Frame Decode Errors
I am writing a RTSP client with Flutter SDK that is also using the Apple VideoToolKit API for the HW decoder. I will refrain from posting code at the moment because I think it would just be easier to explain. I'm using wireshark to inspect contents and it seems parsing is working correctly. My problem is I can't get the data in the right format for the decoder, hence getting OSStatus (Swift) error code -8969. If someone could please clarify the AVCC format along with the inner payload format expected by the decoder, that would be great. My decoding is done for one frame at a time. So for each frame I create a new decoding session. The SPS and PPS value is set to a static value on program startup, and then updated once the server begins sending RTP. I don't parse the sprop-parameter-sets value at the moment, I will add this at a later time. The below buffer will be the resulting AVCC format for a FU-A RTP payload with 3 slices. Please let me know if there is anything I get wrong here. The 4 byte length is big endian representation. When I create the decoding session in Swift, I consider all of this to be 1 sample. [4 byte length][FU identifier slice 1][FU header slice 1][NAL Unit payload slice 1][4 byte length][FU identifier slice 2][FU header slice 2][NAL Unit payload slice 2][4 byte length][FU identifier slice 3][FU header slice 3][NAL Unit payload slice 3] The length is: length = RTSP length field - RTP_HEADER_LEN Where the RTP_HEADER_LEN is equal to 12 bytes. Any guidance appreciated, thank you. Update: I removed the FU header/FU identifier from the buffer, and am now getting 2 successful decodes out of hundreds of frames, but am still getting OSStatus codes: -666, -8969. and -12348. If anyone could explain any of these to me that would be helpful. Thanks.
4
0
1.8k
Dec ’22
H264 videotoolbox wrong decoding
I am trying to decode my h264 stream by using only c++ (not obj-c and swift). I wrote decoding part by following this stackoverflow post. According to status the decoding is successfully completed. But I saw that most of decoded data are zero when check the data. This shoow that decoding is wrong or not completed. I think, I made a mistake at decoding initializing stage. I have added my code to below. VTDecompressionOutputCallbackRecord cb{}; cb.decompressionOutputCallback = &VisionModuleHandler::decode_cb; cb.decompressionOutputRefCon = NULL; CMFormatDescriptionRef format; CMBlockBufferRef block = NULL; CMSampleBufferRef buffer = NULL; VTDecodeInfoFlags flags; OSStatus status; /* Initialize the decoder. */ std::vector<uint8_t> sps(encoded_frame.sps_pps_frame.begin()+4, encoded_frame.sps_pps_frame.begin()+20 ); std::vector<uint8_t> pps(encoded_frame.sps_pps_frame.begin()+24, encoded_frame.sps_pps_frame.end() ); std::size_t sps_pps_size_array [] = {16,5}; const size_t sampleSizeArray {encoded_frame.encoded_frame.size()}; const uint8_t* const parameterSetPointers[2] = {sps.data(), pps.data()}; status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, sps_pps_size_array, 4, &format); if (status != noErr) spdlog::info("Error while decomprasion - 1"); status = VTDecompressionSessionCreate(kCFAllocatorDefault, format, NULL, NULL, &cb, &session); if (status != noErr) spdlog::info("Error while decomprasion - 2"); status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, (void*)(encoded_frame.encoded_frame.data()), encoded_frame.encoded_frame.size(), kCFAllocatorNull, NULL, 0, encoded_frame.encoded_frame.size(), 0, &block); if (status != noErr) spdlog::info("Error while decomprasion - 3"); status = CMSampleBufferCreate(kCFAllocatorDefault, block, true, NULL, NULL, format, 1, 0, NULL, 1, &sampleSizeArray, &buffer); if (status != noErr) spdlog::info("Error while decomprasion - 4"); /* Loop over compressed data; our callback will be called with * each decoded frame buffer. Passed flags make this asynchronous. */ status = VTDecompressionSessionDecodeFrame(session, buffer, 0, NULL, 0 ); if (status != noErr) spdlog::info("Error while decomprasion - 5"); /* Flush in-process frames. */ status = VTDecompressionSessionFinishDelayedFrames(session); if (status != noErr) spdlog::info("Error while decomprasion - 6"); /* Block until our callback has been called with the last frame. */ status = VTDecompressionSessionWaitForAsynchronousFrames(session); if (status != noErr) spdlog::info("Error while decomprasion - 7"); I also added Callback part of decoding. if (imageType == CVPixelBufferGetTypeID()) { auto a = 1; } CVPixelBufferLockBaseAddress(imageBuffer, 0); OSType format = CVPixelBufferGetPixelFormatType(imageBuffer); void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(imageBuffer,0); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); uint8_t *yBuffer = (uint8_t*)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0); size_t yPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0); uint8_t *cbCrBuffer = (uint8_t*)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1); size_t cbCrPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 1); void *srcYpData = malloc(height *yPitch); memcpy(srcYpData, yBuffer, height *yPitch); void *srcCbCrData = malloc(height *cbCrPitch); memcpy(srcCbCrData, cbCrBuffer, height *cbCrPitch); size_t aRgbPitch = width * 4; uint8_t *aRgbBuffer = (uint8_t*)malloc(height *aRgbPitch); memset(aRgbBuffer, 0, height *aRgbPitch); vImage_Buffer srcYp = {srcYpData,height,width,yPitch}; vImage_Buffer srcCbCr = {srcCbCrData,height,width,cbCrPitch}; vImage_Buffer dest = {aRgbBuffer,height,width,aRgbPitch}; vImage_YpCbCrPixelRange pixelRange = {16,128,265,240,235,16,240,16}; vImage_YpCbCrToARGB infoYpCbCrToARGB = {}; vImage_Error error = vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4, &pixelRange, &infoYpCbCrToARGB, kvImage420Yp8_CbCr8, kvImageARGB8888, kvImageNoFlags); uint8_t permuteMap[4] = {0, 1, 2, 3}; error = vImageConvert_420Yp8_CbCr8ToARGB8888(&srcYp, &srcCbCr, &dest, &infoYpCbCrToARGB, permuteMap, 255, kvImageNoFlags); How can I fix this?
2
0
2.4k
Dec ’22
Using AVMutableVideoComposition and AVAssetExportSession to remove the background from video
Hi! Like in the title, I am using AVMutableVideoComposition and AVAssetExportSession to remove the background of the video on export. I'll remove the background with Vision/ML predictions. I am having issues utilizing AVVideoCompositing and AVVideoCompositionInstructionProtocol and AVAssetExportSession. Does anyone have any more up-to-date resources available for examples?
1
1
1.3k
Dec ’22
Video Toolbox got kVTPropertyNotSupportedErr from kVTCompressionPropertyKey_ConstantBitRate in IOS 16.0
Hi All, I'm working on the kVTCompressionPropertyKey_ConstantBitRate under VideoToolBox session. It is available on ios 16.0 and above, but I got the kVTPropertyNotSupportedErr (-12900) on IPhone 11 (IOS 16.0). Can I get more information of what patterns of the hardware encoder's behaviors, such as what kinds of encoders support the kVTCompressionPropertyKey_ConstantBitRate, what kinds of encoders does not support the kVTCompressionPropertyKey_ConstantBitRate? btw, kVTCompressionPropertyKey_ConstantBitRate supports IPhone 8 (IOS 16.1.1), same line of codes. probably, I does not set compression property right, do I? Do we have any demonstrations of how to use the kVTCompressionPropertyKey_ConstantBitRate? Sincerely, Melinda
0
0
1.7k
Dec ’22
Accessing/Returning Data within a Closure that returns Void
Hello, I am new to Swift and have a specific requirement needing to be implemented through the VideoToolbox API. I am using MethodChannel() from Flutter SDK to invoke Swift functions. I need access to the HW video decoder and the plan is to receive encoded frames from flutter application, decode with VideoToolbox API, then return the decoded image back through the MethodChannel(). I am having trouble checking if the session succeeds in decoding a frame. The callback given to VTDecompressionSessionDecodeFrame() via the VTDecompressionOutputCallbackRecord class may only return a void. However, this callback recieves as argument the decoded image buffer and status, which I have no way of accessing at the moment; this is essentially my problem. I tried some method such as creating a variable at a higher level scope but I get compiler error A C function pointer cannot be formed from a closure that captures context. It was also quite frustrating that I simply cannot get the base address of a variable like in C/C++ so I can perform memcpy(), but that is another topic. So my question is, how may I return/access data within a closure that may only return void? Are there any helpful classes to facilitate communication between the closure and function calling VTDecompressionSessionDecodeFrame(). My idea would be to have the calling function wait until the value of the image buffer or status changes. This doesn't sound like a good implementation but it's only what I can think of at the moment. Any help would be appreciated. Thank you.
1
0
1.2k
Dec ’22
Ventura on M1 Mac Minis: ProRes decodes 64RGBALE to all-zero images
When customers update to Ventura on M1 Mac Minis, ProRes decoding using 64RGBALE breaks, supplying all-black images without reporting any error. All continues to work fine on M1 Max and Pro machines. DTS says (as expected) it's a macOS bug; file a bug report. Reported as FB11771668 a month ago, but have heard nothing. Trying to get a little visibility for it, or similar reports.
0
0
2.1k
Dec ’22
AVSampleBufferDisplayLayer freezes UI on init
Hello, I am seeing an issue in my app where creating instances of AVSampleBufferDisplayLayer from AVFoundation is freezing the main UI in some circumstances for more than 5 seconds. Is there any workarounds for this?  I have not been able to reproduce this myself though. I am creating all instances of this class on the main thread. Here is the stack trace that is being logged. 0 libsystem_kernel.dylib:mach_msg_trap (in libsystem_kernel.dylib) + 8 1 MediaToolbox:FigVideoQueueRemoteClient_Create (in MediaToolbox) + 200 2 MediaToolbox:__FigVideoQueueCreateRemote_block_invoke (in MediaToolbox) + 100 3 CoreMedia:FigRPCCreateServerConnectionForObject (in CoreMedia) + 508 4 MediaToolbox:FigVideoQueueCreateRemote (in MediaToolbox) + 300 5 AVFCore:__49-[AVSampleBufferVideoRenderer _createVideoQueue:]_block_invoke (in AVFCore) + 48 6 libdispatch.dylib:_dispatch_client_callout (in libdispatch.dylib) + 20 7 libdispatch.dylib:_dispatch_lane_barrier_sync_invoke_and_complete (in libdispatch.dylib) + 56 8 AVFCore:-[AVSampleBufferVideoRenderer _createVideoQueue:] (in AVFCore) + 164 9 AVFCore:-[AVSampleBufferVideoRenderer init] (in AVFCore) + 528 10 AVFCore:-[AVSampleBufferDisplayLayer init] (in AVFCore) + 280 11 MyApplication:-[PlaybackView initWithFrame:] (in MyApplication) (PlaybackView.m:56) 12 MyApplication:__62-[CustomPlayer setupVideoDisplayLayerWithCompletion:]_block_invoke (in MyApplication) (CustomPlayer.m:730) 13 libdispatch.dylib:_dispatch_call_block_and_release (in libdispatch.dylib) + 32 14 libdispatch.dylib:_dispatch_client_callout (in libdispatch.dylib) + 20 15 libdispatch.dylib:_dispatch_main_queue_drain (in libdispatch.dylib) + 928 16 libdispatch.dylib:_dispatch_main_queue_callback_4CF (in libdispatch.dylib) + 44 17 CoreFoundation:__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ (in CoreFoundation) + 16 18 CoreFoundation:__CFRunLoopRun (in CoreFoundation) + 2532 19 CoreFoundation:CFRunLoopRunSpecific (in CoreFoundation) + 600 20 GraphicsServices:GSEventRunModal (in GraphicsServices) + 164 21 UIKitCore:-[UIApplication _run] (in UIKitCore) + 1100 22 UIKitCore:UIApplicationMain (in UIKitCore) + 364 23 LinkedIn:main (in LinkedIn) (main.m:37) 24 ???:24 ??? 0x000000010877dda4 0x0 + 0
2
0
1.5k
Nov ’22
How to create an video on demand platform for the streaming business?
Looking for the best ott platform provider or solution to launch our own video on demand business with customized features and functionalities, revenue models. We are focusing on movie streaming content to broadcast across the web, IOS, Apple TV, Amazon fire tv and monetize over it. Need suggestion regarding this. Thanks, Advance
4
0
3.0k
Oct ’22
[iOS] Do we have any apis like MediaCodec in Android to get the mediaCapabilties in Swift
is there any class in iOS which returns encoder/ decoder capabilities just like Android MediaCodec/ MediaCodecList.(https://developer.android.com/reference/android/media/MediaCodec) I need to get the fps/profile/level and width /height supported on each profile of h264 and hevc codec. I have found it related to AvCaptureSession, but this may not be correct since we need to support AvPlayer only (and camera is not in the part of the flow.) We got VTIsHardwareDecodeSupported (CMVideoCodecType) -> Bool helpful , but we need more info like the h264/h265 Profiles/levels /Video resolution /fps etc.
0
0
1.8k
Sep ’22
AVEIOP0: ASSERT: CHEVCController_H13S
Everything works fine without encode parameter kVTCompressionPropertyKey_DataRateLimits. The assertion was trigged when using HEVC hardware encoder by setting it to (1.9 * averageBitrate * 8, 2 seconds). The error log shows: default        18:19:53.997736+0800        kernel        AVEIOP0: DPBBuffer ERROR: ref == NULL 4 default        18:19:53.997978+0800        kernel        AVEIOP0: SCRATCH_REG32_RD: addr fffffffff34bc034 value 00000012 default        18:19:53.998025+0800        kernel        AVEIOP0: SCRATCH_REG32_WR: addr fffffffff34bc034 value 40000012 default        18:19:53.998068+0800        kernel        AVEIOP0: ASSERT: ./AppleAVE2FW/kf_controller/H9/CHEVCController_H13S.cpp, 12136: pPicParams->sRef.UV_L0_LSB[me_ref_index] != 0 What's wroing with the hardware encoder? How to do the root cause analyze? Thank you.
1
0
1.4k
Aug ’22
VTCompressionSessionCreate hangs on Iphone 8
I am trying to run the videotoolbox api. It works fine on both emulator (tested Iphone 14 Pro and Iphone SE) and osx Macbook. When running on the Iphone the synchronous method VTCompressionSessionCreate does not return but gets stuck. This is hitting pause in XCode after a few seconds of wait time: The code leading up to this: " let imageBufferAttributes = [         kCVPixelBufferWidthKey: NSNumber(value: width),         kCVPixelBufferHeightKey: NSNumber(value: height),         kCVPixelBufferPixelFormatTypeKey: NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)       ]               let encoderSpecification = [         kVTVideoEncoderSpecification_EnableLowLatencyRateControl: NSNumber(value: true),       ]               let compressionSessionOut = UnsafeMutablePointer<VTCompressionSession?>.allocate(capacity: 1)               // Callback       let encodeCallback: VTCompressionOutputCallback = { outputCallbackRefCon, sourceFrameRefCon, status, infoFlags, sampleBuffer in         if CMSampleBufferDataIsReady(sampleBuffer!) {           let encoder: Encoder = Unmanaged<Encoder>.fromOpaque(sourceFrameRefCon!).takeUnretainedValue()           encoder.writeData(sampleBuffer: sampleBuffer!, infoFlags: infoFlags)         }       }       // Create session       var status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault,                           width: Int32(width),                           height: Int32(height),                           codecType: codecType, //kCMVideoCodecType_H264,                           encoderSpecification: nil,//encoderSpecification as CFDictionary?,                           imageBufferAttributes: imageBufferAttributes as CFDictionary?,                           compressedDataAllocator: kCFAllocatorDefault,                           outputCallback: encodeCallback,                           refcon: Unmanaged.passUnretained(self).toOpaque(),                           compressionSessionOut: compressionSessionOut) " I have searched the web for anyone else seeing this but so far not found anyone. Is my hardware broken? It is an old device (I am waiting for a newer device).
Replies
5
Boosts
1
Views
2.3k
Activity
Mar ’23
VideoToolbox h.264 Encoder: CBR Possible?
Hey folks - I'm a bit clueless about this specific topic, so bear with me please. I make use of the recording/streaming app called OBS Studio on both my Windows and Mac rigs. OBS supports hardware encoding to h.264 for recording and streaming so that you don't have to lean on its internal FFMpeg calls to transcode for you. Big CPU saver there. I note when I'm attempting to stream to YouTube, for instance, that YT reports back that I'm sending all sorts of wonky bitrates. OBS lets me set a "bitrate" and "bitrate limit", which I have to 50Mbit/sec. YouTube will randomly report back that I'm sending it 70Mbit/sec. Or 90+ Mbit/sec. This is brief, but it happens repeatedly. Challenge: streaming services hate VBR. And according to the OBS devs I've talked to, they seem to think that Apple's VT encoders only support VBR. Is that true? Is there any way to force CBR so that my Mac only ever sends 50Mbit/sec, no more, no less? Thanks.
Replies
2
Boosts
0
Views
4.7k
Activity
Mar ’23
h264 encoder with wrong frame order
I use VideoToolbox HW h.264 encoder with M1 MacBook for screen mirroring. I need run encoder with minimal delay mode. I use these values as encoderSpecification kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder:true kVTCompressionPropertyKey_ProfileLevel:kVTProfileLevel_H264_Baseline_AutoLevel kVTCompressionPropertyKey_RealTime:true kVTCompressionPropertyKey_AllowFrameReordering:false I set presentation timestamp with current time. In compressed callback, I got encoded frame with wrong order. [13.930533]encode pts=...13.930511 [13.997633]encode pts=...13.997617 [14.013678]compress callback with pts=...13.997617 dts=...13.930511 [14.023443]compress callback with pts=...13.930511 dts=...13.997617 in[]:log time, pts:presentation timestamp, dts:decode timestamp AllowFrameReordering is not working as I expected. If I need to set other property, please let me know. I also does not like buffering 2 video frames. If you know settings for no buffering frame, please let me know.
Replies
2
Boosts
0
Views
1.9k
Activity
Jan ’23
Frame Decode Errors
I am writing a RTSP client with Flutter SDK that is also using the Apple VideoToolKit API for the HW decoder. I will refrain from posting code at the moment because I think it would just be easier to explain. I'm using wireshark to inspect contents and it seems parsing is working correctly. My problem is I can't get the data in the right format for the decoder, hence getting OSStatus (Swift) error code -8969. If someone could please clarify the AVCC format along with the inner payload format expected by the decoder, that would be great. My decoding is done for one frame at a time. So for each frame I create a new decoding session. The SPS and PPS value is set to a static value on program startup, and then updated once the server begins sending RTP. I don't parse the sprop-parameter-sets value at the moment, I will add this at a later time. The below buffer will be the resulting AVCC format for a FU-A RTP payload with 3 slices. Please let me know if there is anything I get wrong here. The 4 byte length is big endian representation. When I create the decoding session in Swift, I consider all of this to be 1 sample. [4 byte length][FU identifier slice 1][FU header slice 1][NAL Unit payload slice 1][4 byte length][FU identifier slice 2][FU header slice 2][NAL Unit payload slice 2][4 byte length][FU identifier slice 3][FU header slice 3][NAL Unit payload slice 3] The length is: length = RTSP length field - RTP_HEADER_LEN Where the RTP_HEADER_LEN is equal to 12 bytes. Any guidance appreciated, thank you. Update: I removed the FU header/FU identifier from the buffer, and am now getting 2 successful decodes out of hundreds of frames, but am still getting OSStatus codes: -666, -8969. and -12348. If anyone could explain any of these to me that would be helpful. Thanks.
Replies
4
Boosts
0
Views
1.8k
Activity
Dec ’22
H264 videotoolbox wrong decoding
I am trying to decode my h264 stream by using only c++ (not obj-c and swift). I wrote decoding part by following this stackoverflow post. According to status the decoding is successfully completed. But I saw that most of decoded data are zero when check the data. This shoow that decoding is wrong or not completed. I think, I made a mistake at decoding initializing stage. I have added my code to below. VTDecompressionOutputCallbackRecord cb{}; cb.decompressionOutputCallback = &VisionModuleHandler::decode_cb; cb.decompressionOutputRefCon = NULL; CMFormatDescriptionRef format; CMBlockBufferRef block = NULL; CMSampleBufferRef buffer = NULL; VTDecodeInfoFlags flags; OSStatus status; /* Initialize the decoder. */ std::vector<uint8_t> sps(encoded_frame.sps_pps_frame.begin()+4, encoded_frame.sps_pps_frame.begin()+20 ); std::vector<uint8_t> pps(encoded_frame.sps_pps_frame.begin()+24, encoded_frame.sps_pps_frame.end() ); std::size_t sps_pps_size_array [] = {16,5}; const size_t sampleSizeArray {encoded_frame.encoded_frame.size()}; const uint8_t* const parameterSetPointers[2] = {sps.data(), pps.data()}; status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, sps_pps_size_array, 4, &format); if (status != noErr) spdlog::info("Error while decomprasion - 1"); status = VTDecompressionSessionCreate(kCFAllocatorDefault, format, NULL, NULL, &cb, &session); if (status != noErr) spdlog::info("Error while decomprasion - 2"); status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, (void*)(encoded_frame.encoded_frame.data()), encoded_frame.encoded_frame.size(), kCFAllocatorNull, NULL, 0, encoded_frame.encoded_frame.size(), 0, &block); if (status != noErr) spdlog::info("Error while decomprasion - 3"); status = CMSampleBufferCreate(kCFAllocatorDefault, block, true, NULL, NULL, format, 1, 0, NULL, 1, &sampleSizeArray, &buffer); if (status != noErr) spdlog::info("Error while decomprasion - 4"); /* Loop over compressed data; our callback will be called with * each decoded frame buffer. Passed flags make this asynchronous. */ status = VTDecompressionSessionDecodeFrame(session, buffer, 0, NULL, 0 ); if (status != noErr) spdlog::info("Error while decomprasion - 5"); /* Flush in-process frames. */ status = VTDecompressionSessionFinishDelayedFrames(session); if (status != noErr) spdlog::info("Error while decomprasion - 6"); /* Block until our callback has been called with the last frame. */ status = VTDecompressionSessionWaitForAsynchronousFrames(session); if (status != noErr) spdlog::info("Error while decomprasion - 7"); I also added Callback part of decoding. if (imageType == CVPixelBufferGetTypeID()) { auto a = 1; } CVPixelBufferLockBaseAddress(imageBuffer, 0); OSType format = CVPixelBufferGetPixelFormatType(imageBuffer); void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(imageBuffer,0); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); uint8_t *yBuffer = (uint8_t*)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0); size_t yPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0); uint8_t *cbCrBuffer = (uint8_t*)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1); size_t cbCrPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 1); void *srcYpData = malloc(height *yPitch); memcpy(srcYpData, yBuffer, height *yPitch); void *srcCbCrData = malloc(height *cbCrPitch); memcpy(srcCbCrData, cbCrBuffer, height *cbCrPitch); size_t aRgbPitch = width * 4; uint8_t *aRgbBuffer = (uint8_t*)malloc(height *aRgbPitch); memset(aRgbBuffer, 0, height *aRgbPitch); vImage_Buffer srcYp = {srcYpData,height,width,yPitch}; vImage_Buffer srcCbCr = {srcCbCrData,height,width,cbCrPitch}; vImage_Buffer dest = {aRgbBuffer,height,width,aRgbPitch}; vImage_YpCbCrPixelRange pixelRange = {16,128,265,240,235,16,240,16}; vImage_YpCbCrToARGB infoYpCbCrToARGB = {}; vImage_Error error = vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4, &pixelRange, &infoYpCbCrToARGB, kvImage420Yp8_CbCr8, kvImageARGB8888, kvImageNoFlags); uint8_t permuteMap[4] = {0, 1, 2, 3}; error = vImageConvert_420Yp8_CbCr8ToARGB8888(&srcYp, &srcCbCr, &dest, &infoYpCbCrToARGB, permuteMap, 255, kvImageNoFlags); How can I fix this?
Replies
2
Boosts
0
Views
2.4k
Activity
Dec ’22
Using AVMutableVideoComposition and AVAssetExportSession to remove the background from video
Hi! Like in the title, I am using AVMutableVideoComposition and AVAssetExportSession to remove the background of the video on export. I'll remove the background with Vision/ML predictions. I am having issues utilizing AVVideoCompositing and AVVideoCompositionInstructionProtocol and AVAssetExportSession. Does anyone have any more up-to-date resources available for examples?
Replies
1
Boosts
1
Views
1.3k
Activity
Dec ’22
Video Toolbox got kVTPropertyNotSupportedErr from kVTCompressionPropertyKey_ConstantBitRate in IOS 16.0
Hi All, I'm working on the kVTCompressionPropertyKey_ConstantBitRate under VideoToolBox session. It is available on ios 16.0 and above, but I got the kVTPropertyNotSupportedErr (-12900) on IPhone 11 (IOS 16.0). Can I get more information of what patterns of the hardware encoder's behaviors, such as what kinds of encoders support the kVTCompressionPropertyKey_ConstantBitRate, what kinds of encoders does not support the kVTCompressionPropertyKey_ConstantBitRate? btw, kVTCompressionPropertyKey_ConstantBitRate supports IPhone 8 (IOS 16.1.1), same line of codes. probably, I does not set compression property right, do I? Do we have any demonstrations of how to use the kVTCompressionPropertyKey_ConstantBitRate? Sincerely, Melinda
Replies
0
Boosts
0
Views
1.7k
Activity
Dec ’22
Accessing/Returning Data within a Closure that returns Void
Hello, I am new to Swift and have a specific requirement needing to be implemented through the VideoToolbox API. I am using MethodChannel() from Flutter SDK to invoke Swift functions. I need access to the HW video decoder and the plan is to receive encoded frames from flutter application, decode with VideoToolbox API, then return the decoded image back through the MethodChannel(). I am having trouble checking if the session succeeds in decoding a frame. The callback given to VTDecompressionSessionDecodeFrame() via the VTDecompressionOutputCallbackRecord class may only return a void. However, this callback recieves as argument the decoded image buffer and status, which I have no way of accessing at the moment; this is essentially my problem. I tried some method such as creating a variable at a higher level scope but I get compiler error A C function pointer cannot be formed from a closure that captures context. It was also quite frustrating that I simply cannot get the base address of a variable like in C/C++ so I can perform memcpy(), but that is another topic. So my question is, how may I return/access data within a closure that may only return void? Are there any helpful classes to facilitate communication between the closure and function calling VTDecompressionSessionDecodeFrame(). My idea would be to have the calling function wait until the value of the image buffer or status changes. This doesn't sound like a good implementation but it's only what I can think of at the moment. Any help would be appreciated. Thank you.
Replies
1
Boosts
0
Views
1.2k
Activity
Dec ’22
Ventura on M1 Mac Minis: ProRes decodes 64RGBALE to all-zero images
When customers update to Ventura on M1 Mac Minis, ProRes decoding using 64RGBALE breaks, supplying all-black images without reporting any error. All continues to work fine on M1 Max and Pro machines. DTS says (as expected) it's a macOS bug; file a bug report. Reported as FB11771668 a month ago, but have heard nothing. Trying to get a little visibility for it, or similar reports.
Replies
0
Boosts
0
Views
2.1k
Activity
Dec ’22
AVSampleBufferDisplayLayer freezes UI on init
Hello, I am seeing an issue in my app where creating instances of AVSampleBufferDisplayLayer from AVFoundation is freezing the main UI in some circumstances for more than 5 seconds. Is there any workarounds for this?  I have not been able to reproduce this myself though. I am creating all instances of this class on the main thread. Here is the stack trace that is being logged. 0 libsystem_kernel.dylib:mach_msg_trap (in libsystem_kernel.dylib) + 8 1 MediaToolbox:FigVideoQueueRemoteClient_Create (in MediaToolbox) + 200 2 MediaToolbox:__FigVideoQueueCreateRemote_block_invoke (in MediaToolbox) + 100 3 CoreMedia:FigRPCCreateServerConnectionForObject (in CoreMedia) + 508 4 MediaToolbox:FigVideoQueueCreateRemote (in MediaToolbox) + 300 5 AVFCore:__49-[AVSampleBufferVideoRenderer _createVideoQueue:]_block_invoke (in AVFCore) + 48 6 libdispatch.dylib:_dispatch_client_callout (in libdispatch.dylib) + 20 7 libdispatch.dylib:_dispatch_lane_barrier_sync_invoke_and_complete (in libdispatch.dylib) + 56 8 AVFCore:-[AVSampleBufferVideoRenderer _createVideoQueue:] (in AVFCore) + 164 9 AVFCore:-[AVSampleBufferVideoRenderer init] (in AVFCore) + 528 10 AVFCore:-[AVSampleBufferDisplayLayer init] (in AVFCore) + 280 11 MyApplication:-[PlaybackView initWithFrame:] (in MyApplication) (PlaybackView.m:56) 12 MyApplication:__62-[CustomPlayer setupVideoDisplayLayerWithCompletion:]_block_invoke (in MyApplication) (CustomPlayer.m:730) 13 libdispatch.dylib:_dispatch_call_block_and_release (in libdispatch.dylib) + 32 14 libdispatch.dylib:_dispatch_client_callout (in libdispatch.dylib) + 20 15 libdispatch.dylib:_dispatch_main_queue_drain (in libdispatch.dylib) + 928 16 libdispatch.dylib:_dispatch_main_queue_callback_4CF (in libdispatch.dylib) + 44 17 CoreFoundation:__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ (in CoreFoundation) + 16 18 CoreFoundation:__CFRunLoopRun (in CoreFoundation) + 2532 19 CoreFoundation:CFRunLoopRunSpecific (in CoreFoundation) + 600 20 GraphicsServices:GSEventRunModal (in GraphicsServices) + 164 21 UIKitCore:-[UIApplication _run] (in UIKitCore) + 1100 22 UIKitCore:UIApplicationMain (in UIKitCore) + 364 23 LinkedIn:main (in LinkedIn) (main.m:37) 24 ???:24 ??? 0x000000010877dda4 0x0 + 0
Replies
2
Boosts
0
Views
1.5k
Activity
Nov ’22
How to create an video on demand platform for the streaming business?
Looking for the best ott platform provider or solution to launch our own video on demand business with customized features and functionalities, revenue models. We are focusing on movie streaming content to broadcast across the web, IOS, Apple TV, Amazon fire tv and monetize over it. Need suggestion regarding this. Thanks, Advance
Replies
4
Boosts
0
Views
3.0k
Activity
Oct ’22
[iOS] Do we have any apis like MediaCodec in Android to get the mediaCapabilties in Swift
is there any class in iOS which returns encoder/ decoder capabilities just like Android MediaCodec/ MediaCodecList.(https://developer.android.com/reference/android/media/MediaCodec) I need to get the fps/profile/level and width /height supported on each profile of h264 and hevc codec. I have found it related to AvCaptureSession, but this may not be correct since we need to support AvPlayer only (and camera is not in the part of the flow.) We got VTIsHardwareDecodeSupported (CMVideoCodecType) -> Bool helpful , but we need more info like the h264/h265 Profiles/levels /Video resolution /fps etc.
Replies
0
Boosts
0
Views
1.8k
Activity
Sep ’22
AVEIOP0: ASSERT: CHEVCController_H13S
Everything works fine without encode parameter kVTCompressionPropertyKey_DataRateLimits. The assertion was trigged when using HEVC hardware encoder by setting it to (1.9 * averageBitrate * 8, 2 seconds). The error log shows: default        18:19:53.997736+0800        kernel        AVEIOP0: DPBBuffer ERROR: ref == NULL 4 default        18:19:53.997978+0800        kernel        AVEIOP0: SCRATCH_REG32_RD: addr fffffffff34bc034 value 00000012 default        18:19:53.998025+0800        kernel        AVEIOP0: SCRATCH_REG32_WR: addr fffffffff34bc034 value 40000012 default        18:19:53.998068+0800        kernel        AVEIOP0: ASSERT: ./AppleAVE2FW/kf_controller/H9/CHEVCController_H13S.cpp, 12136: pPicParams->sRef.UV_L0_LSB[me_ref_index] != 0 What's wroing with the hardware encoder? How to do the root cause analyze? Thank you.
Replies
1
Boosts
0
Views
1.4k
Activity
Aug ’22