Zsh kills Python process with plenty of available VM

All the documentation I was able to consult says that macOS is designed to use up to all available storage on the startup disk (which is the one I am using since I have only one disk, and the available space aforementioned reflects this) for swapping when physical RAM is not enough.

Sure, that's what the system will do. Strictly speaking, it will actually start warning the user and then automatically terminating processes as it approaches "full", but it will basically use "all" available storage.

However...

Then why is the process killed long before the swapping area is exhausted?

...the fact that the system is willing to use "all" available storage doesn't mean that it should let any random process do that. Every process on the system has its own memory limit (both address space and used pages) enforced by the kernel. I'm not sure what the default limit is...

once about 40 or so...

...however, 40 GB doesn't seem like a terrible default. Keep in mind that the point of the default isn't simply to prevent the drive from filling up, but is really about enforcing "reasonable" behavior. Most processes never get anywhere CLOSE to using 40 GB of memory, so in practice, this limit is a lot closer to "how much memory will the system let a broken process pointlessly leak". From that perspective, 40 GB is extremely generous.

In terms of determining the exact size, os_proc_available_memory() will tell you how far from the limit you actually are and is much easier to use than task_info(). I think getrlimit()/setrlimit() (see the man page for more info) would also work, though raising the limit requires super user.

Thanks for any insight you can share on this issue. Is it a known bug (perhaps with conda/Python environments) or is it expected behaviour?

It is very much expected behaviour.

In contrast, the same process on a Linux machine (basic Python venv here) just keeps swapping, and never gets killed until the swap area is exhausted.

Yes. Well, everyone has made choices they're not proud of.

Is there a way to keep the process from being killed?

The limit itself is raisable. Have you tried using "ulimit()" in the shell? Aside from that, I'm not sure mapped files[1] are tracked through the same limit, so you might be able to map a 50 GB file even though the VM system wouldn't let you allocate 40 GB.

[1] In practice, mapped I/O is why hitting this limit isn't common. Most applications that want to interact with large amounts of RAM also have some interest in preserving whatever it is they're manipulating.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you so much for your reply, now I have a picture of what is going on. Could you share also how to use these functions? The only documentation I could find does not have examples. Say I have, among all, this process running, labelled python3 with PID 33238. I tried writing os_proc_available_memory() in my terminal (bash shell), and all I get is a prompt > awaiting for input. Same with getrlimit and setrlimit. I tried also os_proc_available_memory(33238) etc but I get error messages. The documentation keep mentioning 'the current process' but there are many, how do I run this functions relative to a specific ongoing process?

So, I got some more time to look at this today and need to change my answer a bit. Let me actually jump back to here:

All the documentation I was able to consult says that macOS is designed to use up to all available storage on the startup disk (which is the one I am using since I have only one disk and the available space aforementioned reflects this) for swapping, when physical RAM is not enough.

Did you actually see this in our documentation and, if so, where? If any of our "modern" documentation actually says that we do this, then that's something I'd like to clean up and correct.

So, let me go back to the idea here:

macOS is designed to use up to all available storage on the startup disk

That's the "classic” UNIX system machine swap and, historically, it's how Mac OS X originally worked. However, the downside of this approach is "swap death" in one of two forms:

1. "Soft", meaning the system has slowed down to the point that the user is now no longer willing to use the machine, even though it technically still "works".

2. "Hard", meaning the system outstanding swap "debt" has become so high that the system can no longer make forward progress, as the time required to manipulate the VM system swamps all other activity.

The distinction I'm drawing here is about recoverability— the "soft" state is recoverable, as the machine is still functional enough that the user can shutdown/terminate work, returning the system to normal. Hard is not recoverable, as the system itself has become so buried in VM activity that it can't resolve the issue.

Historically, the slow performance of spinning disk meant that this issue was largely self-regulating, as the machine would be come slower and slower in a relatively "linear" way and the user would then push the machine as much as they could tolerate.

That basic concept is well understood, but what's NOT commonly understood is how the dynamics of those failures have changed as hardware and software have evolved.

However, two major things have changed over time:

1. Increasing CPU power meant that it became feasible to compress VM without obvious performance impact, allowing us to store and retrieve higher volumes of physical memory.

2. SSDs dramatically improved I/O performance, particularly random I/O, allowing the system to "jump around" on physical media in a way it really can't on spinning disks.

Those both dramatically increase the benefit of VM, but they also create new scenarios. Notably:

• SSDs are fundamentally "consumable" devices, which eventually run out of write cycles. Allowing unbounded swap file usage to destroy hardware is obviously not acceptable.

• Compression can slow/delay freeing memory/swap, since freeing memory can require additional memory as the system has to decompress swap so that it can dispose of the freed memory, then recompress the data it still needs.

• The combination of compression and SSD performance makes it possible for the machine to swing VERY suddenly from operating normally into swap death with very little notice or warning.

Expanding on that last point, sequences like this become possible:

• The user uses a high memory application, which builds up significant memory use.

• The user backgrounds that application and moves on to other work for an extended period of time. As a result, "all" of that application’s memory is compressed and streamed out to disk.

• The user switches back to the app and immediately starts trying to interact with "all" of its memory.

Under very high load, that sudden usage swing can actually overload the entire machine, as it's trying to simultaneously "flip" the entire state of the system. Critically, the difference here isn't that it can't happen on a spinning disk (it can), it's that the slower performance of a spinning disk meant that it was far less likely to happen "suddenly".

So, let's go back to here:

however, once about 40 or so swap files are created, for a total of about 40GB of virtual memory occupied

What's actually going on here is that the system has artificially capped the amount of swap space it's willing to use. The size is derived from total RAM (more RAM, higher limit), and I suspect you have a 16 GB machine, as I hit exactly the same limit on my machine. However, the limit isn't tied to the specific size, but it is actually tied to the amount of swap being used. In my test app, I hit the limit at ~45 GB when using rand() to fill memory, but was able to go to ~88 GB when I filled with "1s". Better compression meant more memory.

That then leads back to here:

Is there a way to keep the process from being killed?

No, not when working with VM-allocated memory. The system’s swap file limits are basically fixed, based on total RAM.

However, if you really want to exceed this limitation, the most direct approach would be to allocate 1 or more files, mmap those files into memory, then use that mapped memory instead of system-allocated memory. That bypasses the limit entirely, since the limit is specifically about the system swap file usage, not memory usage. However, be aware that this approach does have the same NAND lifetime issues.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I see, thank you for the explanation. Yes, my machine has 16 GB of RAM and I read about the VMM at https://developer.apple.com/library/archive/documentation/Performance/Conceptual/ManagingMemory/Articles/AboutMemory.html#//apple_ref/doc/uid/20001880-BCICIHAB

Is there a guide for macOS on the steps you describe at the end, that is on how to allocate a swapfile, mmap that swapfile into memory, then use that mapped memory instead of system-allocated memory? I am familiar with dd from /dev/zero etc and the usual declaration of a swapfile by appending on /etc/fstab, but that is in Linux, and perhaps that does not work on macOS...

That's the "classic” UNIX system machine swap

Well, I’d argue that the classic Unix swap involved a swap partition. None of this fancy, new fangled anonymous pager stuff (-:

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Is there a guide for macOS on the steps you describe at the end, that is on how to allocate a swapfile,

Just to be clear, you're not actually creating a "swapfile" as such. Your mapping a file into memory which means, assuming the mapping MAP_SHARED, the VM system will then use that file as the backing store for that memory range. That gives you basically the "same" behavior as swap backed system memory, but it isn't ACTUALLY the same as using true "swap" (for example, you'll be writing the data directly "back" to file, so there won't be any compression or VM level encryption).

mmap that swapfile into memory,

mmap is a standard Unix API, which means we don't really provide specific documentation for it, however, there is an old code snippet here showing how it works. One note I will note is that some of the recommendations there aren't really relevant anymore, particularly any recommendation about limiting mapping size. Those concerns where driven by the limited 32-bit address space, but that's not an issue with 64-bit addressing. In any case, here is a rundown of what's involved:

• Create a file on disk that's a large as you want it to be. Note that you'll want it to be a multiple of 16KB (the page size) so that you get a one-to-one mapping between VM pages and your file.

• open the file.

• pass the file into mmap (note that you'll need to pass in "MAP_SHARED" for to create a mapping the writes to disk).

...and the pointer returned by mmap will be the start address of the file you mapped, which you could then use however you wanted. The man page for mmap does a decent job of covering the detail.

then use that mapped memory instead of system-allocated memory?

So, what mmap return to your app is basically just "memory" that you can use like any other memory. Technically, you could build a malloc replacement that used mmap as it's backing store, however, the kind of code that hits this limitation tends to be working with fairly large allocations anyway, since it's hard to get to 40+ GB if you're only allocating a few KB at a time. Because of that, it's typically easiest to rearchitect those large allocations around mmap while continuing to use malloc for "normal" work.

Well, I’d argue that the classic Unix swap involved a swap partition. None of this fancy, new fangled anonymous pager stuff (-:

I am but a young sapling beneath your ancient oak.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Apologies if I misunderstand things completely, I am no developer so memory management is completely foreign to me. Are the steps you describe involving the creation of a file and mmap it, to be used in the shell, and the output then applied to a process already running (say some python code) or are they supposed to be performed before the python code is run? In the first case, how do I assign this points to the process already running? I assume this will be extra virtual memory on top of the one the VMM has already used for the process. In the second case, does the pointer have to be used within the python script (that is, do I need to modify my python script in a certain way to tap into this file for virtual memory)? Or once I have it, I have to launch the process in a certain way that it uses it? Feel free to refer me to some resource that explains how to map a file to a process, all I could find, manages included, was not helpful and does not go into enough detail for my background. There is no mention of the process' PID, so I am quite confused about how to let the process know that it can use this file via the pointer. Further, once this is set-up, will the process still use the available physical ram or it will only use the file?

One important clarification for my use case, in the eventuality that the script has to be modified via mmap, is that the python process I need to use the mapped file uses a gurobi routine. Only this routine is the memory heavy part. However, this is proprietary software (written in C) which I cannot modify nor have access to. I simply call it through a python API with a command: model.optimize(). Thus I fear, in this case, that mmap is not an option as I do not find any mention of it in the gurobi documentation.

Apologies if I misunderstand things completely. I am no developer, so memory management is completely foreign to me.

OK. So, stepping back for a moment, I think what's helpful to understand is that one of the VM system’s basic "primitive" operations is "designate that a range of address space corresponds to the contents of a file". That process is called "mapping a file" and is how, for example, executable files are loaded into memory so that they can be executed. That is, what part of what "running an executable" actually "means" is "map the contents of that file into memory and start executing at a specific point". Similarly, the basics[1] of swap files are simple "create a file and designate which parts of it correspond to specific memory ranges". Mapping VM address space to file ranges with what mmap() "does". That is, a low-level mach memory function like vm_allocate() (what "malloc" calls within its implementation):

<https://web.mit.edu/darwin/src/modules/xnu/osfmk/man/vm_allocate.html>

...basically means "allocate some address space and back it up with the system’s backing store" (meaning swap files), while mmap() means "allocate some address space and back it up with this file".

[1] Things like encryption and page compression were then later added onto the "basic" process.

Are the steps you describe involving the creation of a file and mmap it, to be used in the shell, and the output then applied to a process already running (say some Python code) or are they supposed to be performed before the Python code is run? In the first case, how do I assign this point to the process already running?

Neither. It's something your Python code does. See:

<https://docs.python.org/3/library/mmap.html>

However, this is proprietary software (written in C) which I cannot modify nor have access to. I simply call it through a Python API with a command: model.optimize(). Thus I fear, in this case, that mmap is not an option as I do not find any mention of it in the gurobi documentation.

So, this goes back to what I described in my earlier section. mmap() doesn't create some kind of "different" memory. Memory is memory. What's actually going on is that "all" memory is in some sense "mapped" and using mmap() simply means that you are directly controlling a process that would otherwise be invisible.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I understand now, thanks for your patience. Unfortunately this means that the only way for this to work is having the Gurobi libraries' code, and incorporate mmap there, because that's where the memory-heavy node-files are created and put in memory. Too bad this is not open-source software, and therefore this is impossible. Is there an alternative to mmap to perhaps squeeze some extra space in the disk? I was wondering if renicing the process to, say, -20, can have some indirect benefits...

I understand now, thanks for your patience. Unfortunately, this means that the only way for this to work is having the Gurobi libraries' code and incorporating mmap there, because that's where the memory-heavy node-files are created and put in memory.

I don't know anything about their product, so it might be worth raising the issue with them.

Too bad this is not open-source software, and therefore this is impossible. Is there an alternative to mmap to perhaps squeeze some extra space in the disk?

No, not really. The limit you’re hitting is actually a system-wide cap enforced by the kernel. You're hitting it because your single process is using an extraordinarily large amount of memory, but you'll see exactly the same failure with more processes using smaller amounts of memory.

I was wondering if renicing the process to, say, -20, can have some indirect benefits...

No, not really. Theoretically, you could free up some additional memory by reducing "total" system activity, but you're using so much memory that I think the difference will be fairly marginal (maybe a few GB?). In theory, there is a boot arg that can raise the limit (see the kernel code here), but I haven't looked into the specifics in any detail.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you, I asked, but unfortunately from Gurobi they confirmed their APIs do not support mmap. I am curious about your last suggestion regarding the boot parameter vm_compression_limit. The default is zero, but I cannot find any documentation about the other possible values, nor what is the corresponding behaviour. Is there any official documentation on it? Is there a command that supports modifying this parameter? On my mac it seems encrypted since on the kernels folder (I actually do not know which kernel is in use, there are multiple ones) all the files display a bunch of random symbols, so it is not as easy as to edit as the Linux kernel boot parameters. I assume there is a command to do it.

Thank you, I asked, but unfortunately from Gurobi they confirmed their APIs do not support mmap. I am curious about your last suggestion regarding the boot parameter vm_compression_limit. The default is zero.

Not exactly. If you look at the code, "0" is actually what you get when the boot arg isn't set at all, in which case the system uses the logic that follows in that function to define its value. Setting "vm_compression_limit=<overcommit pages>" will directly override that value, setting it to whatever you want.

Is there a command that supports modifying this parameter?

The "nvram" command will let you set boot args. See the man page for more details.

but I cannot find any documentation about the other possible values, nor what is the corresponding behaviour. Is there any official documentation on it?

No. If you want to mess with it, my suggestion would be that you write a tool then "eats" large amounts of memory and logging as memory increases. On an idle machine, that will let you see exactly when you hit the limit and your process is terminated.

I've attached an example of this code below; however, it comes with a few warnings:

1. This is test-only code I threw together quickly and is not an example of how "real" code should work.

2. Intentionally pushing the VM system like this isn't really safe. Case in point, if you run multiple copies of this tool at the same time, it's entirely possible you'll hang or panic the kernel. While I've filed a bug on that (r.166329804), this is very much a case of "don't do that".

3. This code calls task_info() to track its own memory usage. That is useful in this particular context, but I would strongly recommend NEVER using this code in a shipping app. Our mach APIs can be much trickier to use than they seem, and I've seen multiple cases of mach APIs (and task_info in particular) introducing catastrophic bugs[1] into shipping, typically due to mach port leaks.

//
//  main.m
//  Mem_Limit_Test
//
//  Created by Kevin Elliott on 12/11/25.
//

#import <Foundation/Foundation.h>

static bool continueEat = true;

void printVMusage()
{
    NSDate* startDate = [NSDate date];
    NSTimeInterval timeLapsed = 0.0;

    task_name_t selfTask = mach_task_self();
    while(1) {
        task_vm_info_data_t t_info;
        mach_msg_type_number_t t_info_count = TASK_VM_INFO_COUNT;
        if (KERN_SUCCESS == task_info(selfTask, TASK_VM_INFO, (task_info_t)&t_info, &t_info_count))
        {
//            NSLog(@"Total: %lu Resident: %lu", t_info.virtual_size, t_info.resident_size);
            double footSizeGB = t_info.phys_footprint/1024/1024;
            double resSizeMB = t_info.resident_size/1024/1024;
            timeLapsed = (startDate.timeIntervalSinceNow*-1.0);
            NSLog(@"Foot: %.4lf(MB) Resident: %.2lf(MB) Time: %l0.6f(s)", footSizeGB, resSizeMB, timeLapsed);
            if(!continueEat){
                NSLog(@"Hanging!");
                while(1) {
                    sleep(10);
                }
            }
            if(0) {
//            if(footSizeGB>= 42000.0000) {
            //if(footSizeGB>= 4494.0000) {
                NSLog(@"Ending Memory");
                continueEat = false;
            }
        }
        sleep(1);
    }
}


void eatmemory(int index);

void *alloc_and_fill_16mb(void) {
    // Allocate 16 MB
    size_t allocSize = 1024 * 4 * 4 * 100;
    unsigned char *buffer = valloc(allocSize);
    if (!buffer) {
        perror("valloc failed");
        return NULL;
    }
    for (size_t i = 0; i < allocSize; i++) {
        buffer[i] = rand() & 0xFF;   // lower 8 bits
//        buffer[i] = 1;   // lower 8 bits
    }

    return buffer;
}


void eatmemory(int index)
{
    
    
    while(continueEat) {
        int* mem = alloc_and_fill_16mb();
        //NSLog(@"%d, %d", mem[303], mem[2999]);
    }
    NSLog(@"End Eat %d", index);
    
}

int main(int argc, const char * argv[]) {
    @autoreleasepool {
        // insert code here...
        srand((unsigned) time(NULL));

        int pid = [[NSProcessInfo processInfo] processIdentifier];
        NSUInteger cpuCount = [[NSProcessInfo processInfo] activeProcessorCount] - 2;

        NSLog(@"PID: %d cpuCount: %d", pid, cpuCount);
        task_vm_info_data_t t_info;
        mach_msg_type_number_t t_info_count = TASK_VM_INFO_COUNT;
        if (KERN_SUCCESS == task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&t_info, &t_info_count))
        {
//            NSLog(@"Total: %lu Resident: %lu", t_info.virtual_size, t_info.resident_size);
            double footSize = t_info.phys_footprint/1024;
            double resSizeMB = t_info.resident_size/1024/1024;

            NSLog(@"Foot: %.4lf(KB) Resident: %.2lf(MB)", footSize, resSizeMB);

        }
        sleep(3);
        NSLog(@"Starting...");
        
        int indexCount = 0;

        while(indexCount < cpuCount){
            [NSThread detachNewThreadWithBlock:^{
                eatmemory(indexCount);
            }];
            indexCount++;
        }
        printVMusage();
    }
    return 0;
}

[1] These bugs have easily been the most interesting bugs in my long career in DTS, and as you can imagine, "interesting to DTS" generally means that weeks or months of time and energy have already been wasted before the problem ever got to us. Friends, don't let friends call into Mach.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you for sharing. The following is an AI generated overview, but if it is correct, it seems that the only thing that vm_compression_limit would change is the compression of physical memory, not virtual memory. Am I interpreting wrong what follows?

"The vm_compression_limit boot argument in macOS accepts integer values that define the maximum percentage of physical memory allowed for compressed memory. The values represent the percentage limit: 0: The default value, which allows the system to determine the appropriate limit dynamically based on system load. 1 through 100: Sets the specific maximum percentage of physical RAM that the compressed memory store is allowed to consume. Setting a specific value overrides the system's dynamic management. Usage Example To set the compression limit to 50%, you would add the following to your boot arguments in your config.plist (for OpenCore users) or through the nvram command in recovery mode (with System Integrity Protection disabled): vm_compression_limit=50 Note that changing this value requires disabling System Integrity Protection (SIP) for it to take effect."

I am trying to find in advance which way it is most likely to pan out me messing with this boot-arg parameter, since I would not want to mess things up for nothing. Essentially the above states that with 100 you get maximum compression (in my case it is allowed to compress the whole 16 GB of ram). But it does not seem that this will change the swapping behaviour, it will merely start a little later (since I have compressed more ram). So once the system started swapping, the vmm will again do its thing.

Thank you for sharing. The following is an AI-generated overview, but if it is correct, it seems that the only thing that vm_compression_limit would change is the compression of physical memory, not virtual memory.

That AI description is a mixed bag of semi-nonsense. As a general warning, this is not an area where I'd trust AI to produce a reliable answer. The AI system is generating its answer by "making up" an answer based on scraping the internet, but the problem is that this doesn't really work when very little conversation/information has been published. Amusingly, if you tell the AI system it's wrong, it will helpfully accept your feedback and start making up new nonsense that's closer to the truth.

In terms of the specific description, there are a few different issues:

• The percentage description is simply wrong. It's true that the default value is based on a percentage of RAM, but the whole "point" of the boot-arg is to override the default by providing a fixed value.

• The description of how compressed memory relates to physical memory and swap is wrong/misleading. Compressed memory exists because, particularly before the introduction of SSDs, the performance gap between storage and CPU had gotten so large that it was faster for the CPU to compress/decompress pages and write/read that smaller volume vs. simply writing/reading pages directly. Putting that more bluntly, if the CPU is going to be stuck twiddling its thumbs waiting on disk I/O, then it's better off using that time to compress memory instead of doing "nothing".

• SSDs did narrow that gap to some degree; however, the performance gap is still quite large, and the write count limitations of SSDs mean that reducing write volume helps extend SSD life as well.

The key point here is that memory compression is part of the "pipeline" that leads toward swap, to the point that all/most of what's actually written out to swap is compressed.

Finally, on this point:

Am I interpreting what follows incorrectly?

The other point that is missing here is that hitting the vm_compression_limit is what actually starts termination behavior you're seeing. Note the commented-out line in the code I posted above:

    for (size_t i = 0; i < allocSize; i++) {
        buffer[i] = rand() & 0xFF;   // lower 8 bits
//        buffer[i] = 1;   // lower 8 bits
    }

If you swap the commented-out lines so that you're filling with a constant value instead of random data, you'll find that the memory the tools are able to use DRAMATICALLY jumps—in my testing it went from ~40GB -> ~80GB. That's because the fixed values are much more compressible, so the system ends up being able to stuff more data into the same real storage "space".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I see, thank you for pointing this out. So it is not a percantage, but an actual number of pages. Could you expand a little on how to interpret <overcommit pages> in your previous answer? How does one find the available range? What does it mean to overcommit pages? Ideally, I would try to get as close as possible to a memory overcommitment scenario, would this correspond to an "infinite" number of overcommitted pages? Is there a way to enter "infinite" in this parameter? Or there is a maximum number, which can change from machine to machine? If I am interpreting the directionality this parameter has to move towards to, in order to get the desired behaviour, I need to retrieve this number, not just compute it roughly via the known 4KB size of a page and the capacity of the disk.

Say the maximum number is 1200 pages. From the documentation, I am supposed to boot in recovery mode, disable SIP, and then run

sudo nvram boot-args="vm_compressor_limit=1200"

and then restart to make the changes effective. Do I need to keep SIP disabled or I can re-enable it after the changes make effect?