Linking against Python shared library to make distribution

Our general advice is that you bundle the version of Python you need with your app. Given that you can’t do that, things are going to get tricky.

Let’s start with an easy question: Is your app sandboxed?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo , it is not

OK. That makes this task feasible.

There are two basic strategies to make this work:

• Load and call the Python framework dynamically.

• Put your Python code into a separate helper tool.

I’ll discuss each in turn.

To load the Python framework dynamically, pass its path to dlopen. To look up symbols in the framework, call dlsym. To call them, cast the result to an appropriate function pointer and call that.

Because you want to support arbitrary copies of the Python framework, you’ll have to disable library validation. See Disable Library Validation Entitlement.

IMPORTANT Disabling library validation makes it harder to pass Gatekeeper. See Resolving Gatekeeper Problems Caused by Dangling Load Command Paths.

The main drawback to this approach is that, if you call a lot of different routines in the Python framework, you need to write a bunch of glue code. You can automate that to some extent, but it’s still a pain.

The alternative is to put all the code that calls the Python framework into a separate tool and then run that tool with dynamic linker environment variables that allow it to find the Python framework in the right place. See the dyld man page for info about those environment variables.

You’ll have to disable library validation in the tool. You’ll also have to explicitly re-enable dynamic linker runtime variable support. See Allow DYLD Environment Variables Entitlement.

I see two potential drawbacks to this approach. The first is that it’ll only work with a specific release of Python. That’s because the Python framework encodes its version into its install name:

% otool -l Python3.framework/Python3 | grep -B 1 -A 2 LC_ID_DYLIB
Load command 4
          cmd LC_ID_DYLIB
      cmdsize 72
         name @rpath/Python3.framework/Versions/3.9/Python3 (offset 24)

See Dynamic Library Identification for more on that.

The second drawback is that you’ll have to add an IPC layer between your app and your tool.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo , thanks for suggestions!

Adding com.apple.security.cs.disable-library-validation resolved my issue. However, I encountered a problem when adding com.apple.security.cs.allow-dyld-environment-variables. The application can't start because SIP complains that "it is from an unidentified developer," but app is definitely signed.

I found a workaround for this issue by removing allow-dyld-environment-variables but keeping disable-library-validation. Instead DYLD_LIBRARY_PATH variable I can create a symlink to the Python library (@executable_path/../lib/libpython3.11.dyld -> PATH_TO_LIBPYTHON). This solution works like a charm.

For both approaches allow-dyld-environment-variables and symlink I using the same binary signed with different entitlements.

$ otool -L xtensa-esp-elf-gdb-3.11
xtensa-esp-elf-gdb-3.11:
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
  @executable_path/../lib/libpython3.11.dylib (compatibility version 3.11.0, current version 3.11.0)
  /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 907.9.0)

I'm curious if there's a way to use DYLD_LIBRARY_PATH without such a symlink workaround?

The application can't start because SIP complains that "it is from an unidentified developer," but app is definitely signed.

This is not SIP but rather Gatekeeper. That is exactly the dangling load command issue I referenced earlier. If you end up disabling library validation then you must find and fix all your dangling load commands.

I can create a symlink to the Python library

Lemme get this straight. Are you saying that you created a symlink in the file system with a name that starts with @executable_path? And is that used at build time? Or runtime?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo , let me explain with examples:

For both cases used the same GDB binary with python support (xtensa-esp-elf-gdb-3.11).

$ otool -L xtensa-esp-elf-gdb-3.11
xtensa-esp-elf-gdb-3.11:
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
  @executable_path/../lib/libpython3.11.dylib (compatibility version 3.11.0, current version 3.11.0)
  /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 907.9.0)

Case 1 (library symlink)

Use entitlements (disable-library-validation):

codesign -d --entitlements - --xml xtensa-esp-elf-gdb-3.11
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.cs.disable-library-validation</key><true/></dict></plist>

Run from executable directory:

mkdir ../lib
ln -s /Library/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib ../lib/libpython3.11.dylib
./xtensa-esp-elf-gdb-3.11

This does work

Case 2 (DYLD_LIBRARY_PATH)

Use entitlements (allow-dyld-environment-variables + disable-library-validation):

$ codesign -d --entitlements - --xml xtensa-esp-elf-gdb-3.11
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.cs.allow-dyld-environment-variables</key><true/><key>com.apple.security.cs.disable-library-validation</key><true/></dict></plist>

Run:

export DYLD_LIBRARY_PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib
./xtensa-esp-elf-gdb-3.11

This does not work. Popup window with message about "unidentified developer". How can I see more details about the issue?

For both cases used the same GDB binary with python support

I mean the same binary but signed with different entitlements

Hmmm, I’m not sure why case 1 works. I’d have expected it to fail.

Taking a step back, Gatekeeper problems that show up when you disable library validation are usually caused by dangling load commands. See Resolving Gatekeeper Problems Caused by Dangling Load Command Paths. Best practice is to:

1. Re-enable library validation. That solves this problem and yields better security.

2. If that’s not possible, fix the dangling load command.

I don’t think either is possible in your case. You have to disable library validation so that you can use a user-supplied Python runtime. And you can’t remove the dangling load command because your executable isn’t bundled.

Hmmm, unless your executable is bundled? Is there a bundle structure surrounding xtensa-esp-elf-gdb-3.11?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo ,

There are no dangling paths in my executable. (/usr/lib/***.dylib and @executable_path/../lib/***.dylib are ok to use)

The reason was found by accident (seems gatekeeper does not provide any debug messages :(.

I download application from a build server. And sometimes I have to run xattr -d com.apple.quarantine ./xtensa-esp-elf-gdb-3.11 to make it works.

I wonder why I have to do this for codesigned and notarized application. Thought that avoiding messages like "from undefined developer" was the target of these actions... What else I have to do to make Gatekeeper happy??

Should I send exactly the same archive to notary service?

Now I send only binaries in zip archive to notary service and than pack application with all required files to tag.gz/tar.xz archive. Should I change the approach?

After notarization succeeds, I distribute tar.gz/tar.xz archives

seems gatekeeper does not provide any debug messages

Fortunately that’s changed with macOS 14 with the introduction of the syspolicy_check tool. One day I’ll update Resolving Trusted Execution Problems to talk about that properly, but today is not that day.

And sometimes I have to run xattr -d com.apple.quarantine

That’s removing quarantine, which effectively means that Gatekeeper isn’t running.

As to how you should test it, the approach I recommend is the one described in Testing a Notarised Product.

When you run your tool, are you running it from within Terminal? Or double clicking it in the Finder? The latter will hit a known Gatekeeper issue, as described in see Tool Blocked by Gatekeeper in Resolving Gatekeeper Problems.

Now I send only binaries in .zip archive to notary service and than pack application with all required files to .tar.gz / .tar.xz archive. Should I change the approach?

There’s nothing fundamentally wrong with that approach. Whether you should change depends on a bunch of factors, like whether you want to support folks double clicking the tool in the Finder.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

When you run your tool, are you running it from within Terminal? Or double clicking it in the Finder?

I'm running the app from the terminal. It is not intended to be clickable from Finder. But I notice that applications that have entitlements are not clickable compared to applications that have no entitlements.

Seems the binary is ok:

spctl -a -t open -vvv --context context:primary-signature  ./xtensa-esp-elf-gdb-3.11
./xtensa-esp-elf-gdb-3.11: accepted
source=Notarized Developer ID
origin=Developer ID Application: ESPRESSIF SYSTEMS (SHANGHAI) CO., LTD. (QWXF6GB4AV)

codesign -vvvv -R="notarized" --check-notarization  ./xtensa-esp-elf-gdb-3.11
./xtensa-esp-elf-gdb-3.11: valid on disk
./xtensa-esp-elf-gdb-3.11: satisfies its Designated Requirement
./xtensa-esp-elf-gdb-3.11: explicit requirement satisfied

Can I install syspolicy_check to Macos 13? Or it requires kernel changes?

I'm running the app from the terminal. It is not intended to be clickable from Finder.

Cool.

Can I install syspolicy_check to macOS 13?

No. It’s only supported on macOS 14.

But I notice that applications that have entitlements are not clickable compared to applications that have no entitlements.

That’s not quite right. There’s two aspects to that:

• Gatekeeper

• Entitlement use

On the Gatekeeper front, the Tool Blocked by Gatekeeper issue I described above is independent of the tool’s entitlements.

On the entitlement front:

• An entitlement is either unrestricted or restricted. See TN3125 Inside Code Signing: Provisioning Profiles.

• Most unrestricted can be applied directly to a tool.

• A restricted entitlement must be authorised a provisioning profile and that makes it clumsy to apply to a tool. See Signing a daemon with a restricted entitlement.

• One specific edge case relates to the sandbox. A sandboxed process can’t be started from Terminal. See Resolving App Sandbox Inheritance Problems.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"