Resolving Trusted Execution Problems

Code Signing General
This thread has been locked by a moderator; it no longer accepts new replies.
DTS Engineer OP
Apple
Created May ’22
Replies 0
Boosts 0
Views 7.9k
Participants 1

I help a lot of developers with macOS trusted execution problems. For example, they might have an app being blocked by Gatekeeper, or an app that crashes on launch with a code signing error.

If you encounter a problem that’s not explained here, start a new thread with the details. Make sure to add relevant tags — like Gatekeeper, Code Signing, and Notarization — so that I see your post.

IMPORTANT macOS 14 has a new tool, syspolicy_check, that was specifically designed to help diagnose problems like this. I plan to update this post once I have more experience with it. In the meantime, however, if you hit a trusted execution problem and it reproduces on macOS 14, please try out syspolicy_check and let us know how that pans out.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Resolving Trusted Execution Problems

macOS supports three software distribution channels:

  • The user downloads an app from the App Store.

  • The user gets a Developer ID-signed program directly from its developer.

  • The user builds programs locally using Apple or third-party developer tools.

The trusted execution system aims to protect users from malicious code. It’s comprised of a number of different subsystems. For example, Gatekeeper strives to ensure that only trusted software runs on a user’s Mac, while XProtect is the platform’s built-in anti-malware technology.

Note To learn more about these technologies, see Apple Platform Security.

If you’re developing software for macOS your goal is to avoid trusted execution entanglements. You want users to install and use your product without taking any special steps. If, for example, you ship an app that’s blocked by Gatekeeper, you’re likely to lose a lot of customers, and your users’ hard-won trust.

Trusted execution problems are rare with Mac App Store apps because the Mac App Store validation process tends to catch things early. This post is primarily focused on Developer ID-signed programs.

Developers who use Xcode encounter fewer trusted execution problems because Xcode takes care of many code signing and packaging chores. If you’re not using Xcode, consider making the switch. If you can’t, consult the following for information on how to structure, sign, and package your code:

Gatekeeper Basics

User-level apps on macOS implement a quarantine system for new downloads. For example, if Safari downloads a zip archive, it quarantines that archive. This involves setting the com.apple.quarantine extended attribute on the file.

Note The com.apple.quarantine extended attribute is not documented as API. If you need to add, check, or remove quarantine from a file programmatically, use the quarantinePropertiesKey property.

User-level unarchiving tools preserve quarantine. To continue the above example, if you double click the quarantined zip archive in the Finder, Archive Utility will unpack the archive and quarantine the resulting files.

If you launch a quarantined app, the system invokes Gatekeeper. Gatekeeper checks the app for problems. If it finds no problems, it asks the user to confirm the launch, just to be sure. If it finds a problem, it displays an alert to the user and prevents them from launching it. The exact wording of this alert varies depending on the specific problem, and from release to release of macOS, but it generally looks like the ones shown in Apple > Support > Safely open apps on your Mac.

The system may run Gatekeeper at other times as well. The exact circumstances under which it runs Gatekeeper is not documented and changes over time. However, running a quarantined app always invokes Gatekeeper.

Unix-y networking tools, like curl and scp, don’t quarantine the files they download. Unix-y unarchiving tools, like tar and unzip, don’t propagate quarantine to the unarchived files.

Confirm the Problem

Trusted execution problems can be tricky to reproduce:

  • You may encounter false negatives, that is, you have a trusted execution problem but you don’t see it during development.

  • You may also encounter false positives, that is, things fail on one specific Mac but otherwise work.

To avoid chasing your own tail, test your product on a fresh Mac, one that’s never seen your product before. The best way to do this is using a VM, restoring to a snapshot between runs. For a concrete example of this, see Testing a Notarised Product.

The most common cause of problems is a Gatekeeper alert saying that it’s blocked your product from running. However, that’s not the only possibility. Before going further, confirm that Gatekeeper is the problem by running your product without quarantine. That is, repeat the steps in Testing a Notarised Product except, in step 2, download your product in a way that doesn’t set quarantine. Then try launching your app. If that launch fails then Gatekeeper is not the problem, or it’s not the only problem!

Note The easiest way to download your app to your test environment without setting quarantine is curl or scp. Alternatively, use xattr to remove the com.apple.quarantine extended attribute from the download before you unpack it. For more information about the xattr tool, see the xattr man page.

Trusted execution problems come in all shapes and sizes. The remaining sections address the most common ones.

App Blocked by Gatekeeper

If your product is an app and it works correctly when not quarantined but is blocked by Gatekeeper when it is, you have a Gatekeeper problem. For advice on how to investigate such issues, see Resolving Gatekeeper Problems.

App Can’t Be Opened

Not all failures to launch are Gatekeeper errors. In some cases the app is just broken. For example:

  • The app’s executable might be missing the x bit set in its file permissions.

  • The app’s executable might be subtly incompatible with the current system. A classic example of this is trying to run a third-party app that contains arm64e code.

    macOS requires that third-party kernel extensions use the arm64e architecture. In other circumstances, stick to arm64 for your shipping products. If you want to test arm64e code locally, see Preparing Your App to Work with Pointer Authentication.

  • The app’s executable might claim restricted entitlements that aren’t authorised by a provisioning profile.

  • Or the app might have some other code signing problem.

Note For more information about provisioning profiles, see TN3125 Inside Code Signing: Provisioning Profiles.

In such cases the system displays an alert saying:

The application “NoExec” can’t be opened.

[[OK]]

Note In macOS 11 this alert was:

You do not have permission to open the application “NoExec”.

Contact your computer or network administrator for assistance.

[[OK]]

which was much more confusing.

A good diagnostic here is to run the app’s executable from Terminal. For example, an app with a missing x bit will fail to run like so:

% NoExec.app/Contents/MacOS/NoExec 
zsh: permission denied: NoExec.app/Contents/MacOS/NoExec

And an app with unauthorised entitlements will be killed by the trusted execution system:

% OverClaim.app/Contents/MacOS/OverClaim 
zsh: killed     OverClaim.app/Contents/MacOS/OverClaim

In some cases running the executable from Terminal will reveal useful diagnostics. For example, if the app references a library that’s not available, the dynamic linker will print a helpful diagnostic:

% MissingLibrary.app/Contents/MacOS/MissingLibrary 
dyld[88394]: Library not loaded: @rpath/CoreWaffleVarnishing.framework/Versions/A/CoreWaffleVarnishing
    …
zsh: abort      MissingLibrary.app/Contents/MacOS/MissingLibrary

Code Signing Crashes on Launch

A code signing crash has the following exception information:

Exception Type:  EXC_CRASH (SIGKILL (Code Signature Invalid))

The most common such crash is a crash on launch. To confirm that, look at the thread backtraces:

Backtrace not available

For steps to debug this, see Resolving Code Signing Crashes on Launch.

One common cause of this problem is running distribution-signed code. Don’t do that! For details on why that’s a bad idea, see Don’t Run App Store Distribution-Signed Code.

Code Signing Crashes After Launch

If your program crashes due to a code signing problem after launch, you might have encountered the issue discussed in Updating Mac Software.

Non-Code Signing Failures After Launch

The hardened runtime enables a number of security checks within a process. Some coding techniques are incompatible with the hardened runtime. If you suspect that your code is incompatible with the hardened runtime, see Resolving Hardened Runtime Incompatibilities.

App Sandbox Inheritance

If you’re creating a product with the App Sandbox enabled and it crashes with a trap within _libsecinit_appsandbox, it’s likely that you’re having App Sandbox inheritance problems. For the details, see Resolving App Sandbox Inheritance Problems.

Library Loading Problem

Most library loading problems have an obvious cause. For example, the library might not be where you expect it, or it might be built with the wrong platform or architecture. However, some library loading problems are caused by the trusted execution system. For the details, see Resolving Library Loading Problems.

Explore the System Log

If none of the above resolves your issue, look in the system log for clues as to what’s gone wrong. Some good keywords to search for include:

  • gk, for Gatekeeper

  • xprotect

  • syspolicy, per the syspolicyd man page

  • cmd, for Mach-O load command oddities

  • amfi, for Apple mobile file integrity, per the amfid man page

  • taskgated, see its taskgated man page

  • yara, discussed in Apple Platform Security

  • ProvisioningProfiles

You may be able to get more useful logging with this command:

% sudo sysctl -w security.mac.amfi.verbose_logging=1

Here’s a log command that I often use when I’m investigating a trusted execution problem and I don’t know here to start:

% log stream --predicate "sender == 'AppleMobileFileIntegrity' or sender == 'AppleSystemPolicy' or process == 'amfid' or process == 'taskgated-helper' or process == 'syspolicyd'"

For general information the system log, see Your Friend the System Log.

Revision History

  • 2024-10-11 Added info about the security.mac.amfi.verbose_logging option. Updated some links to point to official documentation that replaces some older DevForums posts.

  • 2024-01-12 Added a specific command to the Explore the System Log section. Change the syspolicy_check callout to reflect that macOS 14 is no longer in beta. Made minor editorial changes.

  • 2023-06-14 Added a quick call-out to the new syspolicy_check tool.

  • 2022-06-09 Added the Non-Code Signing Failures After Launch section.

  • 2022-06-03 Added a link to Don’t Run App Store Distribution-Signed Code. Fixed the link to TN3125.

  • 2022-05-20 First posted.

Boost
Replies  0
Boosts  0
Views  7.9k
Participants  1
Resolving Trusted Execution Problems
First post date Last post date
 
 
Q