How do I limit C program memory allocation to 1 virtual page on macOS Sonoma?

I need to clarify your requirements here. You wrote:

Are you trying to measure how efficiently it uses physical memory? Or virtual memory? The rest of your post suggests that you’re asking about virtual memory (“how space efficient different Mallocs”) in which case the question of whether the OS swaps out a page is irrelevant.

Can you expand on your overall goal here?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

It’s better to reply as a reply, rather than in the comments; see Quinn’s Top Ten DevForums Tips for this and other titbits.

What I want to ensure that my multiple mallocs I will be calling will only allocate between those virtual addresses.

Do you control the code calling malloc? Or do you want this to apply to all malloc calls made by a process, including calls made by system frameworks that you don’t have the source code for?

Are you trying to do this for a product? Or for an experiment?

That matters because some of the techniques you might want to use are not appropriate if you plan to distribute a product to a wide audience.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

This is just an experiment and specifically for the process I am using, not for all processes on my system. I am the one invoking the malloc calls as well. The goal is just too see how many mallocs of x size can be fit into a range of virtual memory addresses. It is mostly looking to see how much fragmentation is between each malloc call because if you have a fixed range of virtual addresses that this specific process has access to we can see how many mallocs can be called before it goes out of bounds.

OK.

Are you trying measure the behaviour of the system malloc implementation? Or implement your own malloc and measure it’s behaviour?

You can’t force the system malloc implementation to use a specific VM address range but, even if you could, the results are unlikely to be what you’d expect. The system memory allocator is super complex. It uses a variety of suballocation techniques to improve it’s performance — both speed and memory efficiency — in the face of the serious pressure put on it by your typical Mac app. Those techniques mean there you can’t reasonably measure it’s efficiency using the approach you’ve described.

Consider this program:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char **argv) {
    #pragma unused(argc)
    #pragma unused(argv)
    
    void * mem1 = calloc(1, 1);
    void * mem24 = calloc(1, 24);
    void * mem3900 = calloc(1, 3900);
    void * mem256k = calloc(1, 256 * 1024);
    void * mem100m = calloc(1, 100 * 1024 * 1024);
    
    fprintf(stderr, "   mem1: %p\n", mem1);
    fprintf(stderr, "  mem24: %p\n", mem24);
    fprintf(stderr, "mem3900: %p\n", mem3900);
    fprintf(stderr, "mem256k: %p\n", mem256k);
    fprintf(stderr, "mem100m: %p\n", mem100m);

    fprintf(stderr, "pausing, pid: %d\n", (int) getpid());
    pause();

    return EXIT_SUCCESS;
}

I ran this on my Mac and got this:

% ./Test776021
   mem1: 0x600000548030
  mem24: 0x60000074d260
mem3900: 0x136008800
mem256k: 0x138008000
mem100m: 0x125e00000
pausing, pid: 84342

I then ran vmmap against the process. The output is complex, so I’ve done some significant editing to fit it into this post:

% vmmap 84342
…

==== Writable regions for process 84342
REGION TYPE                    START - END         … REGION DETAIL
…
MALLOC_LARGE (reserved)     125e00000-12c200000    … MallocHelperZone_0x104e4c000
…
MALLOC_SMALL                136000000-136800000    … MallocHelperZone_0x104e4c000
MALLOC_MEDIUM               138000000-140000000    … MallocHelperZone_0x104e4c000
…
MALLOC_NANO              600000000000-600020000000 … DefaultMallocZone_0x104e94000
…

You can see:

• mem1 and mem24 were both suballocated from a large chunk of address space labelled MALLOC_NANO. Such chunks of address space are called allocation zones.

• mem3900 was suballocated from a MALLOC_SMALL zone.

• mem256k was suballocated from a MALLOC_MEDIUM zone.

• mem100m was large enough that malloc went straight to the kernel, resulting a MALLOC_LARGE zone that contains a single item.

If you want to measure the efficiency of the system memory allocator, you can use tools like vmmap and heap to do that. But it’s gonna be very hard to compare those results with other allocators. There’s simply no way to configure it to behave in some sort of ‘standard’ way.

If you want to write your own memory allocators and compare the behaviour of each, that’s quite feasible. Lemme know in that case and I can help you get started on that path.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Yeah this is kind of what I was getting as well when I was running Malloc implementations I could get addresses and for the most part it seemed that the malloc allocations the size would be a contiguous range of virtual memory for each malloc but the space between each malloc was vastly different. As for the goal of the benchmark it was looking at different implementations of malloc such as stdlib.h malloc, tcmalloc, and mimalloc... seeing if there was a difference in space efficiency of these approaches. I was pretty sure that maybe what I was looking for was unfeasible or at least from the way I was approaching it this post was mostly me trying to see if I was missing something.

Here is a snippet of how the test looked where report memory would check to see if there was a bounds violation and then turn on a flag i had

while (1) {
       //here would be like where I would replace the implementation
        int *ptr = (int *)malloc(size);
// i read somewhere that sometimes this will get put off somewhere if i did not actively allocate it so i did this to be safe
        for (int i = 0; i < size / sizeof(int); i++) {
            ptr[i] = i; 
            fprintf(fptr2,"J%d:%p\n",i,&ptr[i]);
        }
        /*report memory checks if the amount of memory allocations
           has reached a limit of memory the process could have or violate the bound  */
        report_memory();
        if(flag==1){
            break;
        }
        // Log successful allocation
        fprintf(fptr, "%zu\n", allocation_count);
        allocation_count++;
    }

If your goal is to replace malloc, that’s possible to do, but the difficulty level varies depending on your specific requirements.

If you want to test the allocation pattern of a code base that you control, build that code and then link it directly to your malloc implementation. The static linker will see that your code references malloc and directly connect that reference to your implementation.

For example, if you put this:

#include <stdlib.h>
#include <stdio.h>

int main(int argc, char **argv) {
    #pragma unused(argc)
    #pragma unused(argv)
    void * mem1 = malloc(1);
    void * mem24 = malloc(24);

    fprintf(stderr, "   mem1: %p\n", mem1);
    fprintf(stderr, "  mem24: %p\n", mem24);

    return EXIT_SUCCESS;
}

and this:

#include <assert.h>
#include <mach/mach.h>
#include <stdio.h>

extern void * malloc(size_t size);

extern void * malloc(size_t size) {
    mach_vm_address_t addr = 0;
    kern_return_t kr = mach_vm_allocate(mach_task_self(), &addr, size, VM_FLAGS_ANYWHERE);
    assert(kr == KERN_SUCCESS);
    void * result = (void *) (uintptr_t) addr;
    fprintf(stderr, "did allocate, address: %p\n", result);
    return result;
}

into an Xcode project, running it with product this:

did allocate, address: 0x1000a4000
did allocate, address: 0x1000a8000
   mem1: 0x1000a4000
  mem24: 0x1000a8000

If you want to test code that you don’t control then things get trickier. Apple platforms support a two-level namespace, meaning that when a dynamic library imports a symbol it imports it from a specific library. So, the technique shown above will only affect code that you compile. For example, when fprintf calls malloc it’ll get the system allocator.

To replace malloc for all clients you need to interpose. This isn’t something that Apple supports as API, but it’s fine to use for development tools and for experimentation. I have more info on this in An Apple Library Primer.

IMPORTANT Creating your own memory allocator that’s sufficiently robust to support all clients is quite hard. It needs to be fast, thread-safe, and efficient. Additionally, our memory allocator API is much more extensive than malloc and free. So, if you can get away with using the first technique, you should.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Yeah that was going to be the goal I was just initially just developing the test for generic malloc and then implementing other versions of it and compare results. Thank you for the advice!