dyld not searching in /usr/local/lib (cmd line tools 2397)

Question

It seems impossible to me, but around the time I installed the latest command line tools (xcode-select version 2397) binaries what were built with linkages to @rpath/libfoo.dyld stopped being able to find their dependency. The error looks like this.

dyld[1471]: Symbol not found: _GEOSGeomGetX
  Referenced from: <16DBE67F-CB32-31EE-BCE0-BFB58EEC9740> /Users/pramsey/tmp/capi_indexed_predicate
  Expected in:     <no uuid> unknown
zsh: abort      ./capi_indexed_predicate

If I turn on DYLD_PRINT_SEARCHING, I can see the linker giving up on the rpath entry.

dyld[1501]: find path "@rpath/libgeos_c.1.dylib"
dyld[1501]:   not found: "@rpath/libgeos_c.1.dylib"

I can work around, partially, by setting DYLD_LIBRARY_FALLBACK_PATH to have /usr/local/lib in it, but that is only a partial fix, because SIP will strip that variable for any child processes, which means most of my development and database work is borked.

This seems like a very new quirk, maybe related to the XCode 15 update, at least in my personal time line, has anyone else seen it, or have any clue as to what has changed? (Worth noting, nothing changed in my code, just one day my builds wouldn't run anymore, and that day was the day after I had installed the new commandline tools).

1.9k

Posted by

pwramsey3

Reply

I should add, that the issue is very easy to reproduce (at least on my system). Install a library in /usr/local/lib. Write a test program that links that library. Build it (no errors). Run it, and it fails, with that "Expected in: <no uuid> unknown". In my case I used an example program from the GEOS library, cc -I/usr/local/include capi_indexed_predicate.c -o capi_indexed_predicate -L/usr/local/lib -lgeos_c

—
pwramsey3
There are others seeing this, but so far it seems a small group. https://stackoverflow.com/questions/77137284/dynamic-linker-fails-to-find-symbol-at-runtime-after-updating-to-xcode-15

—
pwramsey3
YOLO, so I just updated to 13.6 and nothing has changed, still cannot resolve libraries in /usr/local/lib.

—
pwramsey3

Answer 1

pwramsey3 wrote:

the issue is very easy to reproduce

So lemme see if I understand your steps here:

Create a dynamic library called libfoo.dylib with an install name of @rpath/libfoo.dylib.
Install that in /usr/local/lib.
Create a command-line tool that calls a symbol exported by that dynamic library.

Is that it?

Because I wouldn’t expect that to work without an extra step, namely, adding an rpath entry to the tool using -rpath /usr/local/lib.

What am I missing here?

ps Just to set the scene, I’m coming at this from the perspective described in Dynamic Library Identification.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by

eskimo

The main thing you are missing is that those steps, as you've correctly laid them out, did in fact result in a runnable executable, right up until installing the XCode 15 command-line tools. Why would that be so? The only thing I can point to is the man page of dyld, which says, regarding DYLD_FALLBACK_LIBRARY_PATH, that "By default, it is set to /usr/local/lib:/usr/lib". Which would explain how a library or executable lacking LC_RPATH entries could still find its dependencies in /usr/local/lib.

—
pwramsey3
So my question is, was this a purposeful change in policy regarding /usr/local/lib for XCode 15? Either in the dyloader no longer searching in /usr/local/lib:/usr/lib as a fallback, or in SIP trimming out that default value, in the same way it trims out user-set values. (This SIP behaviour, by the way, is why I cannot just work around the change by setting DYLD_FALLBACK_LIBRARY_PATH manually. My manual changes get trimmed out by SIP for child processes. I guess I could turn off SIP.)

—
pwramsey3
My observation is also that this previously worked without -rpath. Compiling a simple program with cc -o mycode mycode.c -I/path/to/include -L/path/to/lib/-lstuff (where /path/to/lib/libstuff.dylib exists) previously resulted in an executable that did not need any DYLD_FALLBACK_LIBRARY_PATH or -rpath. I have a binary that works fine and compiled as described above; recompiling the exact same code after an update to XCode 15 results in a binary that cannot find the library at runtime.

—
chad-iris

Add a Comment

Answer 2

I can confirm this behavior, and that it started causing troubles with the recent XCode update. Ran into it with pact-go, which installed a libpact_ffi.dylib under /usr/local/lib with install_name being just libpact_ffi.dylib. Long-winded post here - https://github.com/pact-foundation/pact-go/issues/345 According to dyld man page, this should work, because /usr/local/lib is in the default fallback search paths - and it used to work ok. Have you opened a ticket with Apple already @pwramsey3 ?

Posted by

stan-v

I have not opened a ticket, and I don't really know how to, or even if I am allowed to as a random internet dev rather than an accredited apple dev. Can you?

—
pwramsey3
I was directed to this guide - https://developer.apple.com/forums/thread/712889 - apparently anyone can do it? I can try later today or tomorrow too.

—
stan-v
In the meantime, what worked for us was to set absolute path as install_name for the library - install_name_tool -id /usr/local/lib/my-library.dylib /usr/local/lib/my-library.dylib

—
stan-v

Answer 3

@eskimo In our case there's a minor difference, we don't use rpath, but overall the process is similar:

Create a dynamic library called libfoo.dylib with an install name of libfoo.dylib (no rpath)
Install that in /usr/local/lib.
Create a binary that calls a symbol exported by that dynamic library.

Interestingly, linker works fine, in our case its go compiler - it can find this library without issues even if it's install path is just the file name.

This means linker can search /usr/local/lib by default. However when running the newly built binary, dyld fails to locate it.

This used to work. My understanding is that dyld used to search /usr/local/lib by default. According to man dyld description of DYLD_FALLBACK_LIBRARY_PATH:

If a dylib is not found at its install  path, dyld uses this as a list of directories to search for the
dylib.  By default, it is set to /usr/local/lib:/usr/lib.

If this change is intentional, man page should probably be updated, or if not, it seems like a bug?

Long-worded thread in pact-go github: https://github.com/pact-foundation/pact-go/issues/345

\

Posted by

stan-v

Add a Comment

Answer 4

Just a reminder, if you reply in the comments I’m not notified. See tip 5 in Quinn’s Top Ten DevForums Tips.

My understanding is that dyld used to search /usr/local/lib by default.

That’s understandable, based on what’s in the man page.

I originally suspected that this might be related to the hardened runtime, which radically limits the amount of searching done by the dynamic linker. However, I tried this with the hardened runtime disabled and it doesn’t work their either.

I’m inclined to agree with pwramsey3 that this is a SIP thing. Looking at dynamic linker source, it seems to be controlled by the allowClassicFallbackPaths value which is in turn controlled by AMFI. AMFI stands for Apple mobile file integrity per the amfid man page, and it’s the subsystem responsible for enforcing various restrictions like this.

If this change is intentional, man page should probably be updated … ?

Agree. Please file a bug against it, and post your bug number here, just for the record.

Still, best practice here is to identify your library with either an rpath-relative path or an absolute path, and that’s the approach I recommend going forward.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by

eskimo

Add a Comment

Answer 5

Is there any way to change allowClassicFallbackPaths? Grepping the source I see it shows up via getAMFI() but not where getAMFI() is defined. Seems like flipping that bit would be the cleanest way for people like me to get back the old behaviour for a while, while we change over all our library builds to use full paths on install. For the moment, I have turned off SIP and set DYLD_FALLBACK_LIBRARY_PATH in /etc/zprofile, but that makes me nervous as a full-time solution.

Posted by

pwramsey3

Feedback was submitted on FB13226043

—
pwramsey3

Add a Comment

Answer 6

Feedback was submitted on FB13226043

Thanks.

Is there any way to change allowClassicFallbackPaths?

Once you disable SIP there are all sorts of obscure options to opt in and out of various feature. However, I don’t maintain expertise in those, so I can’t offer any guidance.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by

eskimo

Add a Comment

Answer 7

Feedback was submitted on FB13226043

Oops, I've also submitted mine (didn't see @pwramsey3 submitting theirs), mine is FB13233867

Posted by

stan-v

No worries. We can sort that out on our side.

—
eskimo

Add a Comment

Answer 8

Has there been any update on this issue? Is this going to be fixed or is this something developers should get used to?

Posted by

zap150

Just installed the latest XCode and MacOS 13.6.3 and it's still busted.
```
@(#)PROGRAM:ld  PROJECT:dyld-1022.1
BUILD 13:21:42 Nov 10 2023
```
—
pwramsey3

Add a Comment

Answer 9

I forgot to mention @eskimo in my previous post - could you comment on whether this is going to be fixed or whether this is an intented feature?

Posted by

zap150

Add a Comment

dyld not searching in /usr/local/lib (cmd line tools 2397)

Replies