I have an app that edits photos in your library. When I call
try CIContext().writeHEIFRepresentation(of: editedImage, to: fileURL, format: .RGBA8, colorSpace: originalImage.colorSpace!)
The following is logged to the console:
writeImageAtIndex:1012: ⭕️ ERROR: 'App' is trying to save an opaque image (5712x4284) with 'AlphaLast'. This would unnecessarily increase the file size and will double (!!!) the required memory when decoding the image --> ignoring alpha.
What does that mean and how can I resolve it?
Xcode Version 16.0 (16A242d)
iOS 18.1 (22B82)
Core Image
RSS for tagUse built-in or custom filters to process still and video images using Core Image.
Posts under Core Image tag
49 Posts
Sort by:
Post
Replies
Boosts
Views
Activity
We've recently updated a view which displays photos via a CoreImage chain from a NSOpenGLView subclass to a NSView with a backing CAMetalLayer.
Things are mostly working fine, but we occasionally hit a deadlock involving CALayer and CIMetalCommandQueue. I've made a spindump, it appears none of our code is involved in the locked threads. Despite this, I'm assuming the problem is ours 😅
I saw the mention in the CAMetalLayer documentation about releasing drawables with an @autoreleasepool in drawRect, we have done this and I can't find any places we're retaining a drawable outside drawRect.
https://developer.apple.com/documentation/quartzcore/cametallayer?language=objc
I am seeing this on macOS 15.0.1, M2 Max MacBookPro. We haven't seen it on macOS 14.x but it may be luck as we have not tested much on that OS.
I don't know how to move forward debugging this, any help much appreciated!
The two locking threads in the spindump are MainThread and CI::RenderCompletionQueue.
Thread 0xb3b0f8 DispatchQueue "com.apple.main-thread"(1)
…
CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 364 (QuartzCore + 178484) [0x1a5dba934]
invocation function for block in CA::Context::commit_transaction(CA::Transaction*, double, double*) + 176 (QuartzCore + 1782676) [0x1a5f42394]
-[CALayer(CALayerPrivate) _copyRenderLayer:layerFlags:commitFlags:] + 720 (QuartzCore + 179304) [0x1a5dbac68]
-[NSImage(CALayerSupport) CA_copyRenderValue] + 52 (AppKit + 1517960) [0x1a0fe0988]
-[NSImage CGImageForProposedRect:context:hints:] + 440 (AppKit + 1246368) [0x1a0f9e4a0]
-[NSImage _usingBestRepresentationForRect:context:hints:body:] + 148 (AppKit + 1247980) [0x1a0f9eaec]
__48-[NSImage CGImageForProposedRect:context:hints:]_block_invoke + 80 (AppKit + 1248792) [0x1a0f9ee18]
-[NSCIImageRep CGImageForProposedRect:context:hints:] + 112 (AppKit + 6200292) [0x1a1457be4]
+[CIContext contextWithOptions:] + 40 (CoreImage + 549532) [0x1a8df129c]
-[CIContext initWithOptions:] + 588 (CoreImage + 65744) [0x1a8d7b0d0]
+[CIContext(Internal) internalContextWithMTLDevice:options:] + 76 (CoreImage + 66568) [0x1a8d7b408]
CIMetalCommandQueueCreate + 52 (CoreImage + 66692) [0x1a8d7b484]
-[CaptureMTLDevice newCommandQueue] + 168 (GPUToolsCapture + 130200) [0x1029e7c98]
-[CaptureMTLCommandQueue initWithBaseObject:captureDevice:] + 204 (GPUToolsCapture + 799812) [0x102a8b444]
GTMTLGuestAppClientAddMTLCommandQueueInfo + 108 (GPUToolsCapture + 313572) [0x102a148e4]
__ulock_wait2 + 8 (libsystem_kernel.dylib + 60540) [0x19d24bc7c]
*??? (kernel.release.t6020 + 6102048) [0xfffffe0008cd5c20] (blocked by turnstile waiting for Phocus [11343] [unique pid 1001657] thread 0xb41b08 - part of a deadlock)
and
Thread 0xb41b08 DispatchQueue "CI::RenderCompletionQueue"(535) 1000 samples (1-1000) priority 46 (base 46)
start_wqthread + 8 (libsystem_pthread.dylib + 52464) [0x1035f4cf0]
_pthread_wqthread + 288 (libsystem_pthread.dylib + 20736) [0x1035ed100]
_dispatch_workloop_worker_thread + 580 (libdispatch.dylib + 129956) [0x1026afba4]
_dispatch_root_queue_drain_deferred_wlh + 652 (libdispatch.dylib + 133360) [0x1026b08f0]
_dispatch_lane_invoke + 468 (libdispatch.dylib + 68516) [0x1026a0ba4]
_dispatch_lane_serial_drain + 860 (libdispatch.dylib + 64160) [0x10269faa0]
_dispatch_client_callout + 20 (libdispatch.dylib + 26788) [0x1026968a4]
_dispatch_call_block_and_release + 32 (libdispatch.dylib + 19300) [0x102694b64]
CI::Object::unref() const + 120 (CoreImage + 35360) [0x1a8d73a20]
CI::MetalContext::~MetalContext() + 16 (CoreImage + 192260) [0x1a8d99f04]
CI::MetalContext::~MetalContext() + 236 (CoreImage + 192536) [0x1a8d9a018]
-[CaptureMTLCommandQueue dealloc] + 44 (GPUToolsCapture + 797916) [0x102a8acdc]
GTMTLGuestAppClientRemoveMTLCommandQueueInfo + 236 (GPUToolsCapture + 314240) [0x102a14b80]
GTMTLGuestAppClient_allCaptureObjectsUnsafe + 392 (GPUToolsCapture + 298776) [0x102a10f18]
AllMetalLayers + 64 (GPUToolsCapture + 518224) [0x102a46850]
MakeLayerInfos + 320 (GPUToolsCapture + 518608) [0x102a469d0]
-[CALayer frame] + 88 (QuartzCore + 74624) [0x1a5da1380]
__ulock_wait2 + 8 (libsystem_kernel.dylib + 60540) [0x19d24bc7c]
*??? (kernel.release.t6020 + 6102048) [0xfffffe0008cd5c20] (blocked by turnstile waiting for Phocus [11343] [unique pid 1001657] thread 0xb3b0f8 - part of a deadlock)
I’m working on real-time object detection using YOLOv8, but I only need to detect objects in approximately 40% of the screen area. Is it possible to limit the captureOut method to focus solely on that specific region of the screen?
If this isn’t feasible, I’m considering an approach where the full-screen pixel buffer is captured and then cropped to the target area before running detection. However, I’m concerned about how this might affect real-time performance.
I’d appreciate any insights on how to maintain real-time performance or suggestions for better alternatives. Thank you!
Our application uses Core Image to apply custom CIFilters to still images and video. I'm running into issues when the supplied image is large enough (>4096) that the image is automatically tiled. The simplest of these to describe is a filter that performs various mirroring effects - backwards, upside-down etc.
The implementation portion of the filter provides a sampler (src) and passes this into the kernel with an roiCallback that uses the destRect, inset by -1 in both dimensions:
return [mirrorsKernel applyWithExtent:[src extent] roiCallback:^CGRect(int index, CGRect destRect) { return CGRectInset(destRect, -1, -1); }
arguments:@[src]
];
The kernel is very simple, sampling from the X coordinate equal to the src width - current coordinate:
float4 backwards(sampler image, destination dest)
{
float2 dc = dest.coord();
dc.x = image.size().x - dc.x;
return image.sample(image.transform(dc)));
}
When this runs on an image that is wider than 4096, tiling happens, with the result being that destRect is not the entire image and therefore the resulting output image is incorrect. If the ROI uses [src extent] instead of destRect, the result is correct, but this will lead to serious performance issues when src gets too large.
All of this makes sense to me. What I'd like to know is if there is a way to handle this filter's requirements for sampling from the entire source while still limiting the ROI to maintain performance? I think the answer is probably no within our current structure and performance limits. But I wanted to see if there's anything we're missing.
I am aware that the simple kernel above can be replaced with an affine transform, which is an option for backwards and upside-down mirroring. We have other kernels in this filter that perform mirroring of either half of the source image or one quadrant of the source image. In these cases, I suppose it might be possible (up to a point) to create a custom ROI that is only the portion of the source that is being mirrored. We have not attempted that yet.
Any thoughts/input appreciated, thanks!
Hi everyone,
I've been working with the autoAdjustmentFilters provided by Core Image, which includes filters like CIHighlightShadowAdjust, CIVibrance, and CIToneCurve. However, I’ve noticed that the results differ significantly from the "Auto" enhancement feature in the Photos app. In the Photos app, the Auto function seems to adjust multiple parameters such as contrast, exposure, white balance, highlights, and shadows in a more advanced manner.
Is there an API or a framework available that can replicate the more sophisticated "Auto" adjustments as seen in the Photos app? Or would I need to manually combine filters (like CIExposureAdjust, CIWhitePointAdjust, etc.) to approximate this functionality?
Any insights or recommendations on how to achieve this would be greatly appreciated. Thank you!
In the WWDC 24 session "Use HDR for dynamic image experiences in your app" it's noted this is how you save edits for Adaptive HDR:
SDR + HDR: writeHEIFRepresentation(of: sdrImage, to: url, colorSpace: p3Space, options: [.hdrImage: hdrImage])
SDR + Gain: writeHEIFRepresentation(of: sdrImage, to: url, colorSpace: p3Space, options: [.hdrGainMapImage: gainImage])
This won't compile because the format argument is missing. What format should be used?
In the WWDC 23 session "Support HDR images in your app" RGBAf, RGBAh, and RGBA16, and RGB10 were mentioned but I'm not sure which one to use.
If relevant, I'm editing photos from the user's photo library, so the image was probably taken on iPhone but perhaps not. Thanks!
I am building an app about photos and
I want to create a photo sharing feature like Apple's Photos App.
Please see Steps to Reproduce and attached project.
The current share method has the following issues
The file name of the shared photo changes to “FullSizeRender”.
The creation and update dates of shared photos will change to the date they were edited or shared.
I want to ensure that the following conditions are definitely met
Share the latest edited version.
The creation date should be when the original photo was first created.
How can I improve the code?
STEPS TO REPRODUCE
class PHAssetShareManager {
static func shareAssets(_ assets: [PHAsset], from viewController: UIViewController, sourceView: UIView) {
let manager = PHAssetResourceManager.default()
var filesToShare: [URL] = []
let group = DispatchGroup()
for asset in assets {
group.enter()
getAssetFile(asset, resourceManager: manager) { fileURL in
if let fileURL = fileURL {
filesToShare.append(fileURL)
}
group.leave()
}
}
group.notify(queue: .main) {
self.presentShareSheet(filesToShare, from: viewController, sourceView: sourceView)
}
}
private static func getAssetFile(_ asset: PHAsset, resourceManager: PHAssetResourceManager, completion: @escaping (URL?) -> Void) {
print("getAssetFile")
let resources: [PHAssetResource]
switch asset.mediaType {
case .image:
if asset.mediaSubtypes.contains(.photoLive) {
// let editedResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .fullSizePairedVideo }
// let originalResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .pairedVideo }
let editedResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .fullSizePhoto }
let originalResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .photo }
resources = editedResources.isEmpty ? originalResources : editedResources
} else {
let editedResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .fullSizePhoto }
let originalResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .photo }
resources = editedResources.isEmpty ? originalResources : editedResources
}
case .video:
let editedResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .fullSizeVideo }
let originalResources = PHAssetResource.assetResources(for: asset).filter { $0.type == .video }
resources = editedResources.isEmpty ? originalResources : editedResources
default:
print("Unsupported media type")
completion(nil)
return
}
guard let resource = resources.first else {
print("No resource found")
completion(nil)
return
}
let fileName = resource.originalFilename
let tempDirectoryURL = FileManager.default.temporaryDirectory
let localURL = tempDirectoryURL.appendingPathComponent(fileName)
// Delete existing files and reset cache
if FileManager.default.fileExists(atPath: localURL.path) {
do {
try FileManager.default.removeItem(at: localURL)
} catch {
print("Error removing existing file: \(error)")
}
}
let options = PHAssetResourceRequestOptions()
options.isNetworkAccessAllowed = true
resourceManager.writeData(for: resource, toFile: localURL, options: options) { (error) in
if let error = error {
print("Error writing asset data: \(error)")
completion(nil)
} else {
completion(localURL)
}
}
}
private static func presentShareSheet(_ items: [Any], from viewController: UIViewController, sourceView: UIView) {
print("presentShareSheet")
let activityViewController = UIActivityViewController(activityItems: items, applicationActivities: nil)
if UIDevice.current.userInterfaceIdiom == .pad {
activityViewController.popoverPresentationController?.sourceView = sourceView
activityViewController.popoverPresentationController?.sourceRect = sourceView.bounds
}
viewController.present(activityViewController, animated: true, completion: nil)
}
}```
XCode 16 seems to have an issue with stitchable kernels in Core Image which gives build errors as stated in this question. As a workaround, I selected Metal 3.2 as Metal Language Revision in XCode project. It works on newer devices like iPhone 13 pro and above but metal texture creation fails on older devices like iPhone 11 pro. Is this a known issue and is there a workaround? I tried selecting Metal language revision to 2.4 but the same build errors occur as reported in this question. Here is the code where assertion failure happens on iPhone 11 Pro.
let vertexShader = library.makeFunction(name: "vertexShaderPassthru")
let fragmentShaderYUV = library.makeFunction(name: "fragmentShaderYUV")
let pipelineDescriptorYUV = MTLRenderPipelineDescriptor()
pipelineDescriptorYUV.rasterSampleCount = 1
pipelineDescriptorYUV.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelineDescriptorYUV.depthAttachmentPixelFormat = .invalid
pipelineDescriptorYUV.vertexFunction = vertexShader
pipelineDescriptorYUV.fragmentFunction = fragmentShaderYUV
do {
try pipelineStateYUV = metalDevice?.makeRenderPipelineState(descriptor: pipelineDescriptorYUV)
}
catch {
assertionFailure("Failed creating a render state pipeline. Can't render the texture without one.")
return
}
My app is suddenly broken when I build it with XCode 16. It seems Core Image kernels compilation is broken in XCode 16. Answers on StackOverflow seem to suggest we need to use a downgraded version of Core Image framework as a workaround, but I am not sure if there is a better solution out there.
FYI, I am using [[ stitchable ]] kernels and I see projects having stitchable are the ones showing issue.
air-lld: error: symbol(s) not found for target 'air64_v26-apple-ios17.0.0'
metal: error: air-lld command failed with exit code 1 (use -v to see invocation)
Showing Recent Messages
/Users/Username/Camera4S-Swift/air-lld:1:1: ignoring file '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/CoreImage.framework/CoreImage.metallib', file AIR version (2.7) is bigger than the one of the target being linked (2.6)
SwiftUI [[stichable]] metal shader & CIFilter written in metal extern"C" can't work at the same time
In my project, I used two metal shaders in two ways.
One is link to SwiftUI's modifier .colorEffect(ShaderLibrary.myShader()), which metal shader marked as [[stichable]].
Another one is a custom CIFilter, which kernel been written in external "C" closure.
Because custom CIFilter must add build rules so Xcode can compile it, so I added -fcikernel to Metal Compiler and -cikernel to Metal Linker from Build Settings, just like Apple's document told me that.
But the result is weird, if I add rules, custom CIFilter works, [[stichable]] shader doesn't work. if I delete rules, and comment out code of CIFilter(for avoiding warning), [[stichable]] shader works, but now I can't use my custom CIFilter.
Actually, once these two shaders works well in my project, but when I updated Xcode from 15 to 16, it became weird, the 2 shaders can't exist at same time. Even though I go back to Xcode 15, I can't save them.
I have no idea, please help, thank you.
XCode 16 / iOS 18 on iPhone 14 Pro
I’ve built a iOS camera app that applies many CIFilters to an image captured by the camera. Some of my users have reported that on occasion the images have large parts that are blank, see below:
Frustratingly, I can’t reproduce this myself! Does anyone know what could he causing it, is it a memory issue? I haven’t posted the code as there’s a lot to look over and I’m not sure it would help diagnose it.
Thanks for any pointers.
I was trying to migrate Core Image based code that's rotating an image in a CVPixelBuffer to the newer VTPixelRotationSession from Video Toolbox. Hoping to increase performance.
The original code does:
let rotatedImage = CIImage(cvPixelBuffer: origPixelBuffer).oriented(.left)
context.render(rotatedImage, to: newPixelBuffer)
The new code uses a session:
_ = VTPixelRotationSessionRotateImage(rotationSession, origPixelBuffer, newPixelBuffer)
However I immediately ran into memory limitations, since my code has to be able to run in an iOS extension. It seems VTPixelRotationSessionRotateImage easily lets memory usage spike over the 50MB of allowed memory. While the CIImage based implementation has no such high memory usage at all.
Is this expected? Does the VTPixelRotationSession implementation gain more performance by sacrificing memory? Or is there something I'm overlooking?
I was expecting the VTPixelRotationSession at worst to be on par in terms of memory usage and processing speed compared to CIImage. At this moment it seems VTPixelRotationSession is unusable in extensions.
See also Feedback: FB14977240
Greetings! I have been battling with a bit of a tough issue. My use case is running a pixelwise regression model on a 2D array of images using CIImageProcessorKernel and a custom Metal Shader.
It mostly works great, but the issue that arises is that if the regression calculation in Metal takes too long, an error occurs and the resulting output texture has strange artifacts, for example:
The specific error is:
Error excuting command buffer = Error Domain=MTLCommandBufferErrorDomain Code=1 "Internal Error (0000000e:Internal Error)" UserInfo={NSLocalizedDescription=Internal Error (0000000e:Internal Error), NSUnderlyingError=0x60000320ca20 {Error Domain=IOGPUCommandQueueErrorDomain Code=14 "(null)"}} (com.apple.CoreImage)
There are multiple levels of concurrency: Swift Concurrency calling the Core Image code (which shouldn't have an impact) and of course the Metal command buffer.
Is there anyway to ensure the compute command encoder can complete its work?
Here is the full implementation of my CIImageProcessorKernel subclass:
class ParametricKernel: CIImageProcessorKernel {
static let device = MTLCreateSystemDefaultDevice()!
override class var outputFormat: CIFormat {
return .BGRA8
}
override class func formatForInput(at input: Int32) -> CIFormat {
return .BGRA8
}
override class func process(with inputs: [CIImageProcessorInput]?, arguments: [String : Any]?, output: CIImageProcessorOutput) throws {
guard
let commandBuffer = output.metalCommandBuffer,
let images = arguments?["images"] as? [CGImage],
let mask = arguments?["mask"] as? CGImage,
let fillTime = arguments?["fillTime"] as? CGFloat,
let betaLimit = arguments?["betaLimit"] as? CGFloat,
let alphaLimit = arguments?["alphaLimit"] as? CGFloat,
let errorScaling = arguments?["errorScaling"] as? CGFloat,
let timing = arguments?["timing"],
let TTRThreshold = arguments?["ttrthreshold"] as? CGFloat,
let input = inputs?.first,
let sourceTexture = input.metalTexture,
let destinationTexture = output.metalTexture
else {
return
}
guard let kernelFunction = device.makeDefaultLibrary()?.makeFunction(name: "parametric") else {
return
}
guard let commandEncoder = commandBuffer.makeComputeCommandEncoder() else {
return
}
let imagesTexture = Texture.textureFromImages(images)
let pipelineState = try device.makeComputePipelineState(function: kernelFunction)
commandEncoder.setComputePipelineState(pipelineState)
commandEncoder.setTexture(imagesTexture, index: 0)
let maskTexture = Texture.textureFromImages([mask])
commandEncoder.setTexture(maskTexture, index: 1)
commandEncoder.setTexture(destinationTexture, index: 2)
var errorScalingFloat = Float(errorScaling)
let errorBuffer = device.makeBuffer(bytes: &errorScalingFloat, length: MemoryLayout<Float>.size, options: [])
commandEncoder.setBuffer(errorBuffer, offset: 0, index: 1)
// Other buffers omitted....
let threadsPerThreadgroup = MTLSizeMake(16, 16, 1)
let width = Int(ceil(Float(sourceTexture.width) / Float(threadsPerThreadgroup.width)))
let height = Int(ceil(Float(sourceTexture.height) / Float(threadsPerThreadgroup.height)))
let threadGroupCount = MTLSizeMake(width, height, 1)
commandEncoder.dispatchThreadgroups(threadGroupCount, threadsPerThreadgroup: threadsPerThreadgroup)
commandEncoder.endEncoding()
}
}
I have a 3х3 Matrix which I need to apply to UIImage and save it in Documents folder. I successfully converted the 3x3 Matrix (represented as [[Double]]) to CATrasform3D and then I have broken my head with trying to figure out how to apply it to UIImage.
The only property where I can I apply it is UIView(or UIImageView in case with working with UIImage) transform property. But it has nothing to do with UIImage itself. I can't save the UIImage from transformed the UIImageView with all the transformations.
And all the CoreGraphic methods (like concatenate for CGContext) only work with affine transformations which not suits for me.
Please give me a hint what direction I should look.
Does Apple has native methods or I have to use 3rd party frameworks for this functionality?
Essentially, I'm trying to find the most straightforward/simple way to outline an Image with varying contours. The intention is similar to the way iMessage allows you to add an outline to a sticker. The "goal" in the example is simply the input image on top of the outline.
Hey all 👋🏼
We're currently working on a video processing project using the Vision framework (face, body and hand pose detection), and We've encountered a couple of errors that I need help with. We are on Xcode 16 Beta 3, testing on an iPhone 14 Pro running iOS 18 beta.
The error messages are as follows:
[LOG_ERROR] /Library/Caches/com.apple.xbs/Sources/MediaAnalysis/VideoProcessing/VCPHumanPoseImageRequest.mm[85]: code 18,446,744,073,709,551,598
encountered an unexpected condition: *** -[__NSArrayM insertObject:atIndex:]: object cannot be nil
What we've tried:
Debugging: I’ve tried stepping through the code, but the errors occur before I can gather any meaningful insights.
Searching Documentation: Looked through Apple’s developer documentation and forums but couldn’t find anything related to these specific error codes.
Nil Check: Added checks to ensure objects are not nil before inserting them into arrays, but the error persists.
Here are my questions:
Has anyone encountered similar errors with the Vision framework, specifically related to VCPHumanPoseImageRequest and NSArray operations?
Is there any known issue or bug in the version of the framework I might be using? Could it also be related to the beta?
Are there any additional debug steps or logging mechanisms I can implement to narrow down the cause?
Any suggestions on how to handle nil objects more effectively in this context?
I would greatly appreciate any insights or suggestions you might have. Thank you in advance for your assistance!
Thanks all!
I have an IOSurface and I want to turn that into a CIImage. However, the constructor of CIImage takes a IOSurfaceRef instead of a IOSurface.
On most platforms, this is not an issue because the two types are toll-free bridgeable... except for Mac Catalyst, where this fails.
I observed the same back in Xcode 13 on macOS. But there I could force-cast the IOSurface to a IOSurfaceRef:
let image = CIImage(ioSurface: surface as! IOSurfaceRef)
This cast fails at runtime on Catalyst.
I found that unsafeBitCast(surface, to: IOSurfaceRef.self) actually works on Catalyst, but it feels very wrong.
Am I missing something? Why aren't the types bridgeable on Catalyst?
Also, there should ideally be an init for CIImage that takes an IOSurface instead of a ref.
I’m creating a objective C command-line utility to encode RAW image sequences to ProRes 4444, but I’m encountering, blocky compression artifacts in the ProRes 4444 video output.
To test the integrity of the image data before encoding to ProRes, I added a snippet in my encoding function that saves a 16-bit PNG before encoding to ProRes and the PNG looks perfect, I can see all detail in every part of the image dynamic range.
Here’s a comparison between the 16-bit PNG(on the right) and the ProRes 4444 output. (on the left)
As a further test, I re-encoded the ‘test PNG’ to ProRes 4444 using DaVinci Resolve, and the ProRes4444 output video from Resolve doesn’t have any blocky compression artifacts. Looks identical.
In short, this is what the utility does:
Unpacks the 12-bit raw data into 16-bit values. After unpacking, the raw data is debayered to convert it into a standard color image format (BGR) using OpenCV.
Scale the debayered pixel values from their original 12-bit depth to fit into a 16-bit range. Up to this point everything is fine and confirmed by saving 16bit PNGs.
The images are encoded to ProRes 4444 using the AVFoundation framework.
The pixel buffers are created and managed using dictionary method with ‘kCVPixelFormatType_64RGBALE’.
I need help figuring this out, I’m a real novice when it comes to AVfoundation/encoding to ProRes.
See relevant parts of my 'encodeToProRes' function:
void encodeToProRes(const std::string &outputPath, const std::vector<std::string> &rawPaths, const std::string &proResFlavor) {
NSError *error = nil;
NSURL *url = [NSURL fileURLWithPath:[NSString stringWithUTF8String:outputPath.c_str()]];
AVAssetWriter *assetWriter = [AVAssetWriter assetWriterWithURL:url fileType:AVFileTypeQuickTimeMovie error:&error];
if (error) {
std::cerr << "Error creating AVAssetWriter: " << error.localizedDescription.UTF8String << std::endl;
return;
}
// Load the first image to get the dimensions
std::cout << "Debayering the first image to get dimensions..." << std::endl;
Mat firstImage;
int width = 5320;
int height = 3900;
if (!debayer_image(rawPaths[0], firstImage, width, height)) {
std::cerr << "Error debayering the first image" << std::endl;
return;
}
width = firstImage.cols;
height = firstImage.rows;
// Save the first frame as a PNG 16-bit image for validation
std::string pngFilePath = outputPath + "_frame1.png";
if (!imwrite(pngFilePath, firstImage)) {
std::cerr << "Error: Failed to save the first frame as a PNG image" << std::endl;
} else {
std::cout << "First frame saved as PNG: " << pngFilePath << std::endl;
}
NSString *codecKey = nil;
if (proResFlavor == "4444") {
codecKey = AVVideoCodecTypeAppleProRes4444;
} else if (proResFlavor == "422HQ") {
codecKey = AVVideoCodecTypeAppleProRes422HQ;
} else if (proResFlavor == "422") {
codecKey = AVVideoCodecTypeAppleProRes422;
} else if (proResFlavor == "LT") {
codecKey = AVVideoCodecTypeAppleProRes422LT;
} else {
std::cerr << "Error: Invalid ProRes flavor specified: " << proResFlavor << std::endl;
return;
}
NSDictionary *outputSettings = @{
AVVideoCodecKey: codecKey,
AVVideoWidthKey: @(width),
AVVideoHeightKey: @(height)
};
AVAssetWriterInput *videoInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:outputSettings];
videoInput.expectsMediaDataInRealTime = YES;
NSDictionary *pixelBufferAttributes = @{
(id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_64RGBALE),
(id)kCVPixelBufferWidthKey: @(width),
(id)kCVPixelBufferHeightKey: @(height)
};
AVAssetWriterInputPixelBufferAdaptor *adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:videoInput sourcePixelBufferAttributes:pixelBufferAttributes];
...
[assetWriter startSessionAtSourceTime:kCMTimeZero];
CMTime frameDuration = CMTimeMake(1, 24); // Frame rate of 24 fps
int numFrames = static_cast<int>(rawPaths.size());
...
// Encoding thread
std::thread encoderThread([&]() {
int frameIndex = 0;
std::vector<CVPixelBufferRef> pixelBufferBuffer;
while (frameIndex < numFrames) {
std::unique_lock<std::mutex> lock(queueMutex);
queueCondVar.wait(lock, [&]() { return !frameQueue.empty() || debayeringFinished; });
if (!frameQueue.empty()) {
auto [index, debayeredImage] = frameQueue.front();
frameQueue.pop();
lock.unlock();
if (index == frameIndex) {
cv::Mat rgbaImage;
cv::cvtColor(debayeredImage, rgbaImage, cv::COLOR_BGR2RGBA);
CVPixelBufferRef pixelBuffer = NULL;
CVReturn result = CVPixelBufferPoolCreatePixelBuffer(NULL, adaptor.pixelBufferPool, &pixelBuffer);
if (result != kCVReturnSuccess) {
std::cerr << "Error: Could not create pixel buffer" << std::endl;
dispatch_group_leave(dispatchGroup);
return;
}
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pixelBuffer);
for (int row = 0; row < height; ++row) {
memcpy(static_cast<uint8_t*>(pxdata) + row * CVPixelBufferGetBytesPerRow(pixelBuffer),
rgbaImage.ptr(row),
width * 8);
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
pixelBufferBuffer.push_back(pixelBuffer);
...
Thanks very much!
I'm using CIDepthBlurEffect to create a portrait mode effect on a rendered image. The effect is working as expected however I want to create the "bokeh ball" effect which is seen in the photos app. I see that the filter has a "inputShape" input of type NSString, however the documents do not specify what value this should be.
Any pointers are help is greatly apprecaited.
I feel like I'm missing something really simple. I've got the simplest possible CIKernel, it looks like this:
extern "C" float4 Simple(coreimage::sampler s) {
float2 current = s.coord();
float2 anotherCoord = float2(current.x + 1.0, current.y);
float4 sample = s.sample(anotherCoord); // s.sample(current) works fine
return sample;
}
It's (in my mind) incrementing the x position of the sampler by 1 and sampling the neighboring pixel. What I get in practice is a bunch of banded garbage (pictured below.) The sampler seems to be pretty much undocumented, so I have no idea whether I'm incrementing by the right amount to advance one pixel. The weird banding is still present if I clamp anootherCoord to s.extent() but it behaves normally if I sample s.coord() unchanged. I'm trying to write a box blur that samples / averages neighboring pixels and am completely blocked by this. What am I missing?