AVAssetWriter leading to huge kernel_task memory usage

I am currently working on a macOS app which will be creating very large video files with up to an hour of content. However, generating the images and adding them to AVAssetWriter leads to VTDecoderXPCService using 16+ GB of memory and the kernel-task using 40+ GB (the max I saw was 105GB).

It seems like the generated video is not streamed onto the disk but rather written to memory for it to be written to disk all at once. How can I force it to stream the data to disk while the encoding is happening?

Btw. my app itself consistently needs around 300MB of memory, so I don't think I have a memory leak here.

Here is the relevant code:

Code Block
func analyse()
{
self.videoWritter = try! AVAssetWriter(outputURL: outputVideo, fileType: AVFileType.mp4)
let writeSettings: [String: Any] = [
AVVideoCodecKey: AVVideoCodecType.h264,
AVVideoWidthKey: videoSize.width,
AVVideoHeightKey: videoSize.height,
AVVideoCompressionPropertiesKey: [
AVVideoAverageBitRateKey: 10_000_000,
]
]
self.videoWritter!.movieFragmentInterval = CMTimeMake(value: 60, timescale: 1)
self.frameInput = AVAssetWriterInput(mediaType: AVMediaType.video, outputSettings: writeSettings)
self.frameInput?.expectsMediaDataInRealTime = true
self.videoWritter!.add(self.frameInput!)
if self.videoWritter!.startWriting() == false {
print("Could not write file: \(self.videoWritter!.error!)")
return
}
}
func writeFrame(frame: Frame)
{
/* some more code to determine the correct frame presentation time stamp */
let newSampleBuffer = self.setTimeStamp(frame.sampleBuffer, newTime: self.nextFrameStartTime!)
self.frameInput!.append(newSampleBuffer)
/* some more code */
}
func setTimeStamp(_ sample: CMSampleBuffer, newTime: CMTime) -> CMSampleBuffer {
var timing: CMSampleTimingInfo = CMSampleTimingInfo(
duration: CMTime.zero,
presentationTimeStamp: newTime,
decodeTimeStamp: CMTime.zero
)
var newSampleBuffer: CMSampleBuffer?
CMSampleBufferCreateCopyWithNewTiming(
allocator: kCFAllocatorDefault,
sampleBuffer: sample,
sampleTimingEntryCount: 1,
sampleTimingArray: &timing,
sampleBufferOut: &newSampleBuffer
)
return newSampleBuffer!
}


My specs:
MacBook Pro 2018
32GB Memory
macOS Big Sur 11.1
Answered by nbde in 654244022
The culprits were actually different ones and not in the code I posted.

1.) The kernel_task:

The following code contains the memory leak:
Code Block
let croppedImage = frame.image.cropped(to: CGRect(
x: Int(Int(videoSize.width) / 2 - (self.RECOGNITION_WIDTH / 2)),
y: Int(self.videoSize.height - CGFloat(self.RECOGNITION_HEIGHT)),
width: self.RECOGNITION_WIDTH,
height: self.RECOGNITION_HEIGHT
)


Apparently CIImage.cropped(to:) leads to the original CIImage not being cleaned up. According to my Google search I should have called CGImageRelease(:) but that is not contained in Swift 5.1 since Core Foundation Objects are now memory managed. (At least that is the Xcode error message.)

So I convert the image to a CGImage, crop that one and turn it into a CIImage. I know this is not efficient, but for me that's fine. So here is the new code:
Code Block
import Foundation
import CoreImage
extension CIContext
{
func cropWithoutMemoryLeak(_ image: CIImage, to rect: CGRect) -> CIImage
{
let newImage = self.createCGImage(image, from: rect)
return CIImage(cgImage: newImage!)
}
}


2.) The VTDecoderXPCService:

I was able to reliably remove the memory leak by commenting out this line, which obviously defeats the point of creating a new video:
Code Block
self.frameInput!.append(newSampleBuffer)


After trying some more stuff I removed some of my debug print statements which I had throughout my code. And all the sudden the leak was gone...? 🤷‍♀️ So I don't really know what the problem was. Removing the debug lines was most certainly a correlation but no causation. I guess it triggered a recompile of previously cached files but admittedly I have no idea, what the problem was.

AVAssetWriter writes frames as you append them, but it may take longer to write each frame than it takes you to append each frame. Looking at your code, you don't appear to be checking AVAssetWriterInput.readyForMoreMediaData before appending frames. Updating your code to append frames only when the input is ready may help your memory usage.
Accepted Answer
The culprits were actually different ones and not in the code I posted.

1.) The kernel_task:

The following code contains the memory leak:
Code Block
let croppedImage = frame.image.cropped(to: CGRect(
x: Int(Int(videoSize.width) / 2 - (self.RECOGNITION_WIDTH / 2)),
y: Int(self.videoSize.height - CGFloat(self.RECOGNITION_HEIGHT)),
width: self.RECOGNITION_WIDTH,
height: self.RECOGNITION_HEIGHT
)


Apparently CIImage.cropped(to:) leads to the original CIImage not being cleaned up. According to my Google search I should have called CGImageRelease(:) but that is not contained in Swift 5.1 since Core Foundation Objects are now memory managed. (At least that is the Xcode error message.)

So I convert the image to a CGImage, crop that one and turn it into a CIImage. I know this is not efficient, but for me that's fine. So here is the new code:
Code Block
import Foundation
import CoreImage
extension CIContext
{
func cropWithoutMemoryLeak(_ image: CIImage, to rect: CGRect) -> CIImage
{
let newImage = self.createCGImage(image, from: rect)
return CIImage(cgImage: newImage!)
}
}


2.) The VTDecoderXPCService:

I was able to reliably remove the memory leak by commenting out this line, which obviously defeats the point of creating a new video:
Code Block
self.frameInput!.append(newSampleBuffer)


After trying some more stuff I removed some of my debug print statements which I had throughout my code. And all the sudden the leak was gone...? 🤷‍♀️ So I don't really know what the problem was. Removing the debug lines was most certainly a correlation but no causation. I guess it triggered a recompile of previously cached files but admittedly I have no idea, what the problem was.

I'm glad to hear you've solved or gotten around the issues you are seeing. If you haven't already, please file a bug (using the feedback assistant) about the apparent CIImage memory leak.

I was facing this problem a lot.My CPU usage would reach above 1000 percent and most of it would be by Kernel_task. Whenever we connect second display or use heavy applications ,it requires more gpu/cpu power so our computer heats up more. If the temperature rises too high, Mac Os will run a dummy process "Kernel_task" to reserve cpu usage but keep it idle to reduce load of cpu and protect the computer from burning. As the temperature lowers it will release the resources. The problem in my case was accumulation of dust in the CPU and GPU fan.It had probably sucked it through the side vents. I cleaned it and it works like breeze. I even changed the thermal paste on my CPU and GPU as it had become hard over time and excessive heating. Hope this helps.

Hey - a bit unrelated to the memory leak, but in my case calling AVAssetWriter.startWriting() takes around 3 seconds to complete on a modern iPhone 14 - is there any way to speed that up? See my post: https://developer.apple.com/forums/thread/741942

Thanks!

AVAssetWriter leading to huge kernel_task memory usage
 
 
Q