I have a werid case that shouldn't happen according to https://forums.developer.apple.com/forums/thread/706390
I have an audio unit which runs in FCP and I want it to launch a sandboxed app as a child process. If I sign the child app with just "com.apple.security.app-sandbox" entitlement it crashes with SYSCALL_SET_PROFILE error. According to the article referenced above: "This indicates that the process tried to setup its sandbox profile but that failed, in this case because it already has a sandbox profile." This makes sense because audio units run in a sandboxed environment (in AUHostingService process).
So I added "com.apple.security.inherit" to the entitlements plist and now I get "Process is not in an inherited sandbox." error. According to the article referenced above: "Another cause of a trap within _libsecinit_appsandbox is when a nonsandboxed process runs another program as a child process and that other program’s executable has the com.apple.security.app-sandbox and com.apple.security.inherit entitlements. That is, the child process wants to inherit its sandbox from its parent but there’s nothing to inherit."
And this doesn't make sense at all. The first error indicates the child process is trying to create a sandboxed environment within a parent sandboxed environment while the second error indicates there's no a parent sandboxed environment...
I specifically checked the child process has "com.apple.security.app-sandbox" and "com.apple.security.inherit" entitlements only.
If I remove all entitlements from the child process it launches and runs fine from the audio unit plugin. And if I remove "com.apple.security.inherit" but leave "com.apple.security.app-sandbox" I can successfully launch the app in standalone mode (in Finder).
For the testing puroses I use a simple Hello World desktop application generated by XCode (Obj-C).
Does anybody have an idea what can be the reason for such a weird behavior?
Ah, that’s quite interesting. I wrote Resolving App Sandbox Inheritance Problems on the assumption that you’d always be running either non-sandboxed or in the App Sandbox. That’s not the case here, and hence the lack of a proper explanation.
macOS contains a general-purpose sandboxing facility. That facility is used by a number of different subsystems. App Sandbox is one of those subsystems, but there are many others. For example, MAC — discussed in On File System Permissions — is implemented using the some core sandboxing infrastructure.
Your audio unit is being hosted by AUHostingService. That’s yet another subsystem that uses the general-purpose sandboxing facility. It’s hard to confirm that, but this is one hint:
% nm /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC.xpc/Contents/MacOS/AUHostingServiceXPC | grep sandbox
U _sandbox_init_with_parameters
sandbox_init_with_parameters is an internal variant [1] of sandbox_init that moves the process into a custom sandbox. This sandbox is not the App Sandbox, and thus it doesn’t follow the rules outlined in Resolving App Sandbox Inheritance Problems.
If I remove all entitlements from the child process it launches and runs fine from the audio unit plugin.
Ah, interesting. Clearly AUHostingService’s is not blocking you from creating a child process, which is good.
My understanding is that audio units can’t ship on the App Store. Given that, signing it with no entitlements should be fine. Child processes always inherit their sandbox from their parent. The com.apple.security.inherit entitlement only exists to use in combination with com.apple.security.app-sandbox, and that only has to be there to make the App Store ingestion process happy.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
[1] Keep in mind that this is an implementation detail. See this post for more on background on this topic.