Hello,
We are developing a multimedia routing platform written in Rust and uses gstreamer 1.20. We are targeting running on Mac Minis (older intel and newer M1/2/3/... w/ 8GB ram) using macOS 14.6.1
I have profiled memory usage using XCode instruments with the allocation tool, stack and heap memory is very stable once the pipelines are up and running.
There are between 50 to 100 incoming RTSP streams with multiple webrtc connections, so lots of network and memory bandwidth is being used.
However, we eventually see real memory usage increasing in Activity Monitor along with memory pressure increasing, but the heap/stack usage is constant in instruments, so we do not understand this behavior. Page fragmentation is a possibility, but have not been able to prove this with instruments.
Please see attached image.You can see that 10-minute run had a total of approx 4.3 GB of allocations, but only 50.17MB persistent.
Eventually we see kernel panics in either userspace watchdog timeout: no successful checkins from WindowServer (2 induced crashes) in 120 second or apcie[2:lan-1gb]::handleCompletionTimeoutInterrupt: completion timeout which I believe are caused by high system load and the kernel becoming unresponsive while the kernel is doing page compressions. We tested running with je-malloc for a while, but the kernel panics still occur.
We have multiple kernel panic recordings available, but they are too large to upload here. We are also having multiple kernel panics per day while running this application.
Any suggestions on how to prevent these kernel panics? If the system is out of memory, shouldn't our application crash with an out-of-memory and the kernel should NOT panic?
Thanks,
Jeremy Prater