We're interested in adopting App Sandbox in an app distributed outside of the Mac App Store. However, we're hitting a bit of a roadblock and it doesn't seem like either of the techniques described in that post can be used in a reasonable way.
For background, this is a third-party launcher for a cross-platform Java game that, among other things, makes it easier for users to mod the game. Users generally download mods as .jar files and place them in a certain directory. In some cases, these mods contain native dynamic libraries (e.g. a .dylib) as part of their code. In general, the .dylib is extracted from the contents of the .jar to some temporary location, loaded, and then deleted once the game closes (the exact details, like the actual temporary location, depends on the mod).
App Sandbox greatly interests us in this case because it can limit the damage that a compromised mod could do, and in my testing the functionality of most mods still works with it enabled. However, sandboxed apps quarantine every file they write to by default. Unfortunately, most mods are created by individual developers who don't notarize their libraries (their mods are generally cross-platform, and they're likely just using third-party code that they bundle with the mod but don't sign or notarize). [1] This means that a mod that loads a dynamic library as described above triggers Gatekeeper as described in the documentation if the app is sandboxed, but does not if the sandbox is disabled.
Even worse, a user often can't bypass the warning even if they trust the mod because the extracted library is usually a temporary file, and generally is deleted after the failure (which usually causes the game to crash and thus close). By the time they try to approve the code in System Settings, the file is gone (and even if they could approve it, this approval wouldn't stick next time they launch the game).
In theory it would work to use an unsandboxed XPC service to remove the quarantine and let the libraries through. However, this is easier said than done. We don't control the mods' code or how they go about loading whatever code they need, which limits what we can do.
[1] And in some cases, people like to play old versions of the game with old mods, and the versions they're using might've been released before notarization was even a thing.
The closest thing I can think of to a solution is injecting code into the Java process that runs code to call out to the XPC service to remove the quarantine before a library loads (e.g. before any calls to dlopen using dyld interposition). A prototype I have... works... but this seems really flimsy, I've read that interposition isn't meant to be used in non-dev tools, and if there's a better solution I'd certainly prefer that over this.
Other things we've tried have significant downsides:
com.apple.security.files.user-selected.executable requires user selection in a file picker, and seems to be more blunt than just allowing libraries/plugins which might lead to a sandbox escape [2]
Adding the app to the "Developer Tools" section in System Settings > Privacy & Security allows the libraries to load automatically, but requires users to add the app manually and also sounds like it would make a sandbox escape very easy [2]
Oh, and I also submitted an enhancement request for an entitlement/similar that would allow these libraries to load (FB13795828) but it was returned as "no plans to address" (which honestly wasn't that surprising).
[2] My understanding is that if a sandboxed process loads libraries, the library code would still be confined by the sandbox because it's still running in the sandboxed process. But if a sandboxed process can write and open a non-quarantined app, that app would not be within the confines of the sandbox. So basically we want to somehow allow the libraries to load but not allow standalone executables to run outside the sandbox.
In general the game and almost all popular mods I've tested work with App Sandbox enabled, except for this Gatekeeper snag. It would be a shame to completely abandon App Sandbox for this reason if everything else can be made to work.
This situation seems not super common, but documentation does say
When your sandboxed app launches for the first time, macOS creates a sandbox container on the file system (in ~/Library/Containers) and associates it with your app. Your app has full read and write access to its sandbox container, and can run programs located there as well.
which leaves me wondering whether the Gatekeeper prompt is even intended behavior since the libraries are in the sandbox container and written by the app. (By the way, my testing of the claim that apps can run programs in their sandbox container didn't seem to confirm what the documentation said, even without quarantine - FB15963761). Though, given the other documentation page I linked above which more directly references Gatekeeper and quarantined plug-ins, I doubt this is a bug.
I suppose the final question is, is this just a situation where App Sandbox won't work (at least in any supported way)? Or is there perhaps some technique we're missing?