mprotect fails after MacOS 11.2 for PROT_EXEC pages on Apple Silicon

Hi,

After updating to MacOS 11.2 I noticed that attempting to call mprotect to remove write access on pages that were mapped using PROT_EXEC results in a permission denied error on Apple Silicon. In prior versions of Big Sur, I did not receive this behavior.

Is this a new restriction added in the 11.2 update, and if so, what would be the correct way to implement write protecting a previously RWX page?

A small test program demonstrating what I am trying is shown below:

Code Block
#include <stdio.h>
#include <sys/mman.h>
int main(int argc, const char * argv[]) {
int size = 4096;
int map_flags = MAP_ANON | MAP_PRIVATE | MAP_JIT;
void* mem = mmap(nullptr, size, PROT_READ | PROT_WRITE | PROT_EXEC, map_flags, -1, 0);
if (mem == MAP_FAILED||mem==nullptr)
printf("Failed to allocate executable memory");
if (mprotect(mem, size, PROT_READ | PROT_EXEC) != 0)
printf("WriteProtectMemory failed!\nmprotect: %s\n", strerror(errno));
return 0;
}

This program outputs the below text when ran:

Code Block
WriteProtectMemory failed!
mprotect: Permission denied


Replies

I'm seeing a similar failure. QEMU is setting up guard pages with PROT_NONE after each translation buffer and Big Sur 11.2 doesn't allow to revoke permissions from the guard page while leaving the rest of the mmaped region as is. Another affected project is nodejs: https://github.com/nodejs/node/issues/37061#issuecomment-774175983
I did more exhaustive tests and it looks like the mmap + mprotect behavior on Apple Silicon has been changed between 11.1 -> 11.2 as follows:
  • before and on 11.1, any protection bit combination for mmap(w/MAP_JIT flag) and following mprotect call for the allocated region are succeeded.

  • on 11.2, if PROT_WRITE + PROT_EXEC are specified on mmap(MAP_JIT), then any mprotect operations on the allocated page are failing with "Permission denied".

Here is a test code and a result on my M1 Mac Mini with macOS 11.2.
https://gist.github.com/hikalium/75ae822466ee4da13cbbe486498a191f
My friend has the same machine with macOS 11.0 and they confirmed that mprotect call succeeded on all patterns, which differs from the behavior on 11.2.
You should file a bug on this so we can document the details (or fix the bug if I'm wrong), but I think the change here is actually a natural side effect of the requirements documented here:

Porting Just-In-Time Compilers to Apple Silicon

By definition, "pthread_jit_write_protect_np" is basically superseding the "official" protection state of the underlying pages. I suspect that API never tried to account for the "original" protection level, so the mprotect(RX) call didn't actually do what it was supposed to do, in which case the error is telling you want's actually going on.

Kevin Elliott
DTS CoreOS/Hardware
Thanks for the reply Kevin,

The bug has been filled a while ago: FB8994773

The problematic transition that doesn't work anymore is RWX->NONE for some pages in the allocated MAP_JIT region. It became a problem for QEMU and v8 JS engine. QEMU uses PROT_NONE for guard pages in MAP_JIT region and v8 marks memory with PROT_NONE for sake of memory reclamation.

Roman