App with JIT doesn't work anymore in app bundle and notarization on Apple Silicon

Question

Created Nov ’20

Replies 1

Boosts 0

Views 2k

Participants 1

I'm maintaining the snapshots of DOSBox, a DOS emulator. I'm codesigning these snapshots and also lately began to notarize them. For macOS 10.15 I had to add the entitlement to allow-unsigned-executable-memory, or the notarized app would crash when entering dynamic (JIT) mode.
Now there is a patch to enable the dynamic core for ARM64 and this works nicely when I use the built binary on the Apple silicon (DTK).
But when I make the app bundle it crashes when it enters JIT.
Of course I instantly added the entitlements com.apple.security.cs.allow-jit and just for making sure also com.apple.security.cs.disable-executable-page-protection. codesigned and notarized it but it still crashes with

Code Block CODE SIGNING: 16291[dosbox] vm_map_enter: curprot cannot be write+execute. failing

Now I'm wondering if it's our code that trips over something even though jit, unsigned ex memory and page protection are all set, or whether I'm running into a quirk of the DTK and that works on the actual M1 machines.

To recap, the binary works fine, the app bundle with hardened runtime and following entitlements crashes:

Code Block 	<key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.disable-executable-page-protection</key>
    <true/>
    <key>com.apple.security.automation.apple-events</key>
    <true/>

Answer 1

Dominus OP

Nov ’20

it was a multiple entitlements problem after all, we also had to set sandbox to false (com.apple.security.app-sandbox) and also com.apple.security.cs.disable-library-validation to true, because we are creating a tmpfile when we go into dynamic core.
But unfortunately, we run into another problem:
To test whether it works at all, we hacked our code to have a fixed tmpfile

Code Block memfd = open("/tmp/DOSBox31233", O_RDWR|O_CREAT|O_EXCL,0777);
if (memfd != -1) unlink(tmpfile);

(with the caveat, that the tempfile didn't get deleted on its own until the next reboot, so unless we delete ourselves the next time our app would crash)

Our non hacked code looks like this

Code Block strcpy(tmpfile, "/tmp/DOSBoxXXXXXX");
mktemp(tmpfile);
memfd = open(tmpfile, O_RDWR|O_CREAT|O_EXCL,0777);
LOG_MSG("open result for %s: %d", tmpfile, memfd);
if (memfd != -1) unlink(tmpfile);

But this crashes always with this message in console.app

Code Block ASP: Library load (/private/tmp/DOSBoxV4mAUO) rejected: library load disallowed by system policy

Any idea why THIS doesn't work?