I have code that has worked for many years for writing ProRes files, and it is now failing on the new M1 Max MacBook. Specifically, if I construct buffers with the pixel type "kCVPixelFormatType_64ARGB", after a few frames of writing, the pixel buffer pool becomes nil. This code works just fine on non Max processors (Intel and base M1 natively).
Here's a sample main that demonstrates the problem. Am I doing something wrong here?
// main.m
// TestProresWriting
//
#import <Foundation/Foundation.h>
#import <AVFoundation/AVFoundation.h>
int main(int argc, const char * argv[]) {
@autoreleasepool {
int timescale = 24;
int width = 1920;
int height = 1080;
NSURL *url = [NSURL URLWithString:@"file:///Users/diftil/TempData/testfile.mov"];
NSLog(@"Output file = %@", [url absoluteURL]);
NSFileManager *fileManager = [NSFileManager defaultManager];
NSError *error = nil;
[fileManager removeItemAtURL:url error:&error];
// Set up the writer
AVAssetWriter *trackWriter = [[AVAssetWriter alloc] initWithURL:url
fileType:AVFileTypeQuickTimeMovie
error:&error];
// Set up the track
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecTypeAppleProRes4444, AVVideoCodecKey,
[NSNumber numberWithInt:width], AVVideoWidthKey,
[NSNumber numberWithInt:height], AVVideoHeightKey,
nil];
AVAssetWriterInput *track = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings];
// Set up the adapter
NSDictionary *attributes = [NSDictionary
dictionaryWithObjects:
[NSArray arrayWithObjects:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_64ARGB], // This pixel type causes problems on M1 Max, but works on everything else
[NSNumber numberWithUnsignedInt:width],[NSNumber numberWithUnsignedInt:height],
nil]
forKeys:
[NSArray arrayWithObjects:(NSString *)kCVPixelBufferPixelFormatTypeKey,
(NSString*)kCVPixelBufferWidthKey, (NSString*)kCVPixelBufferHeightKey,
nil]];
/*
NSDictionary *attributes = [NSDictionary
dictionaryWithObjects:
[NSArray arrayWithObjects:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32ARGB], // This pixel type works on M1 Max
[NSNumber numberWithUnsignedInt:width],[NSNumber numberWithUnsignedInt:height],
nil]
forKeys:
[NSArray arrayWithObjects:(NSString *)kCVPixelBufferPixelFormatTypeKey,
(NSString*)kCVPixelBufferWidthKey, (NSString*)kCVPixelBufferHeightKey,
nil]];
*/
AVAssetWriterInputPixelBufferAdaptor *pixelBufferAdaptor = [AVAssetWriterInputPixelBufferAdaptor
assetWriterInputPixelBufferAdaptorWithAssetWriterInput:track
sourcePixelBufferAttributes:attributes];
// Add the track and start writing
[trackWriter addInput:track];
[trackWriter startWriting];
CMTime startTime = CMTimeMake(0, timescale);
[trackWriter startSessionAtSourceTime:startTime];
while(!track.readyForMoreMediaData);
int frameTime = 0;
CVPixelBufferRef frameBuffer = NULL;
for (int i = 0; i < 100; i++)
{
NSLog(@"Frame %@", [NSString stringWithFormat:@"%d", i]);
CVPixelBufferPoolRef PixelBufferPool = pixelBufferAdaptor.pixelBufferPool;
if (PixelBufferPool == nil)
{
NSLog(@"PixelBufferPool is invalid.");
exit(1);
}
CVReturn ret = CVPixelBufferPoolCreatePixelBuffer(nil, PixelBufferPool, &frameBuffer);
if (ret != kCVReturnSuccess)
{
NSLog(@"Error creating framebuffer from pool");
exit(1);
}
CVPixelBufferLockBaseAddress(frameBuffer, 0);
// This is where we would put image data into the buffer. Nothing right now.
CVPixelBufferUnlockBaseAddress(frameBuffer, 0);
while(!track.readyForMoreMediaData);
CMTime presentationTime = CMTimeMake(frameTime+(i*timescale), timescale);
BOOL result = [pixelBufferAdaptor appendPixelBuffer:frameBuffer
withPresentationTime:presentationTime];
if (result == NO)
{
NSLog(@"Error appending to track.");
exit(1);
}
CVPixelBufferRelease(frameBuffer);
}
// Close everything
if ( trackWriter.status == AVAssetWriterStatusWriting)
[track markAsFinished];
NSLog(@"Completed.");
}
return 0;
}
It's been a couple of months since I posted this, and I wanted to try to summarize where we are with this issue. First, I think it's established that this is a bug (my report is FB9757381), and it's related to M1 devices with ProRes hardware acceleration (the Pro and Max variants). The specific bug is that the M1 Pro and Max implementations of the AVFoundation architecture will crash unexpectedly (and without prior warning) when buffers with pixel types kCVPixelFormatType_64ARGB are written. Other pixel types may also fail or behave unexpectedly (there are reports of alpha channel problems), but I haven't exhaustively tested them.
A workaround proposed by the helpful @MirrorMan indicates that on all M1 chips (including Max and Pro), the pixel type kCVPixelFormatType_64RGBALE works without error, but it fails on Intel architectures. As a workaround, it's suboptimal because it requires querying the underlying silicon type and choosing a different pixel type, which is contrary to how we ought to write code on this platform. Also, it's not obvious that if we switched to the workaround pixel format that it would keep working in a future macOS update. In the absence of guidance from Apple, this workaround isn't viable.
This is complicated by the API itself. In the documentation for the CoreVideo type constants, it is indicated that not all types are supported, but there's no way to know which ones are or aren't.
As a developer, I'm stuck. I have an application I support that runs everywhere except on the new MacBook Pro computers, where it crashes. I'm not able to give my clients an estimate on when the program will work. It would be very helpful to me if I could get an idea of Apple's thinking about this issue. For instance, does Apple agree this is a bug, and if so, is there a timeline for a fix? If it's not a bug and it's something I'm doing wrong, I'd like to know that too. If we are to use a workaround, that would be good to know as well.
I know it's normal policy that bugs are not acknowledged or commented upon, but I am asking for some help here. This is a case of new hardware and a somewhat vague API creating a stickier than normal situation, and I'd be very grateful for some feedback.
Hi, at the moment, your feedback item is classified as a bug. However, the team still needs to investigate the issue to confirm this. Unfortunately, we don't give ETAs/timelines for bug fixes. However, your bug hasn't been forgotten about and it's been assigned to an engineer to investigate.
I really appreciate the response. I'll keep watching the releases and keep my fingers crossed. If you get any more info that you're able to share, that'd be great too. Thanks!