Missing librairies in /usr/lib on Big Sur?

To solve a dependency tangle on an app, I’m trying to write a simple command line tool that would display all the dependencies of a given app or library, and output it in a format suitable for post-processing by graphviz. The idea here is to collect and lay out the output of the otool -L utility, recursively called on all the dependencies of the target app/lib.

Unfortunately, when I try otool -L with, say, otool itself, I get this:

Code Block shell
Dev > otool -L /usr/bin/otool
/usr/bin/otool:
/usr/lib/libxcselect.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)

Fine. But now:

Code Block shell
otool -L /usr/lib/libxcselect.dylib
/Library/Developer/CommandLineTools/usr/bin/objdump: error: '/usr/lib/libxcselect.dylib': No such file or directory

Oops. Indeed, /usr/lib seems mostly empty, and most of what lies inside are links on missing (I assume: invisible) libs.

So my question is: where are all the libs gone, and it is possible to bring them back to the surface?
Post not yet marked as solved Up vote post of Vingt-Cent Down vote post of Vingt-Cent
23k views

Replies

Maybe review the WWDC videos and/or post in a beta forum.

All of these libraries have moved into the shared cache. For more details, search the macOS Big Sur 11 Beta 3 Release Notes macOS Big Sur 11.0.1 Release Notes for 62986286.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

previous link is broken

 macOS Big Sur 11 Beta 3 Release Notesfor 62986286.

any other info available?

THX

previous link is broken

This morphed into the main macOS Big Sur 11.0.1 Release Notes.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
I am running into this problem when trying to use gfortran (which should be a subset of gcc ... but in fact is not in /usr/bin

eskimo: I assume you are refering to this paragraph:

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic
linker cache of all system-provided libraries. As part of this change,
copies of dynamic libraries are no longer present on the filesystem.
Code that attempts to check for dynamic library presence by looking for a
file at a path or enumerating a directory will fail. Instead, check for
library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)


But not sure how to interpret it... where is this 'cache'???

Doing, for example a locate libSystem.dylib

Code Block
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/tvOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/watchOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/DriverKit/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/tvOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/watchOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/System/DriverKit/usr/lib/libSystem.dylib

But the directory /System/Volumes/Update/mnt1/ is empty....
And I assume the others don't apply to a Big Sur on an older (2.2 GHz Quad-Core Intel Core i7) mac

Bottom line, which directory should I include to find

I am running into this problem when trying to use gfortran

OK.

I assume you are refering to this paragraph

Yes.

which directory should I include to find

That kinda depends on the capabilities of your toolchain. To understand this I need to explain some backstory…

Way back with the original release of Mac OS X we didn’t support SDKs. When you installed the developer tools it would lay down headers at absolute locations on your current system — for example, the <pcap/pcap.h> header would be placed at /usr/include/pcap/pcap.h — and tools would look for headers at those absolute locations. The same thing would happen for libraries, except the linker would look in /usr/lib.

So far, so Unix.

However, this design did not meet Apple’s needs. The most obvious problem here is that you can’t have multiple SDKs installed. This is a hassle because you can’t, for example, build your production product against the production headers and libraries while also building your in-development product against the next beta headers and libraries.

Apple solved this by creating SDKs. An SDK includes all the headers and libraries needed to build for a specific version of the OS. For example, the macOS 11.1 SDK included in Xcode 12.4 includes all the headers and libraries needed to build for macOS 11.1. Additionally, our toolchains support the concept of a deployment target, which allows you to build with the macOS 11.1 SDK but still run on, say, macOS 10.15.

Apple also used this SDK feature to support multiple platforms. Xcode now contains a macOS SDK, an iOS SDK, and so on.

One problem with these SDKs is that they’re rather large. Once significant contribution to that size was the libraries. Each SDK would include a copy of the libraries necessary to link for that platfrom. However, this is nonsense when you think about it. There’s no point shipping an iOS library on macOS because it’s just a bunch of code that can’t ever execute [1]. The only thing that the linker needs from these libraries is a list of symbols. So in recent Xcode releases we’ve replaced the libraries in your SDKs with stub libraries [2]. That is, there’s no longer a libpcap.dylib library in the SDK, but rather a much smaller libpcap.tbd stub library [3]. It’s a text file (.tbd stands for text based description) so feel free to open it up and take a look.

Given the above reality, the libraries in /usr/lib are no longer needed:
  • Developer tools should be looking at the stub libraries in the appropriate SDK.

  • The runtime doesn’t use these libraries because they’ve been rolled into the dynamic linker shared cache [4].

And that’s why they were removed.



This is fine if you’re using Xcode, or Apple’s command-line tools, because they know how to look inside an SDK for headers and stub libraries. If you’re using third-party tools then you need to consult the support resources for those tools to find out how they’ve adjusted to this new reality. In your case the critical point is the linker. If your third-party tools use the Apple linker then you should just be able to point the tools at the usr/include directory within the appropriate SDK. Our linker will automatically use any stub libraries that it finds.

For example, if you have Xcode installed in the normal place and you’re targeting macOS then the correct directory is:

Code Block
/Applications/
Xcode.app/
Contents/
Developer/
Platforms/
MacOSX.platform/
Developer/
SDKs/
MacOSX.sdk/
usr/
lib


OTOH, if your third-party tools are using their own linker then you’ll have to determine whether that linker has been updated to use stub libraries. I can’t help you with that, alas.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, nowadays we allow users to run iOS apps on Apple silicon but that’s besides the point (-:

[2] This is a term, and a concept, that originated on traditional Mac OS.

[3] Technically, both of these files are symlinks to the corresponding libpcap.A.*** files.

[4] This term is slightly off. Historically this was actually a cache that the system would build from its working set of libraries and frameworks. On macOS 11 this is no longer a cache because the original files are no longer present on the system. Rather, this shared cache is authored by Apple and distributed in toto via the software update mechanism.
Add a Comment

Hi! I've a problem with something regarding this and found this explanation. So, I've a question, if a program made for a previous version of OSX is trying to use /usr/lib/libSystem.B.dylib it will fail, won't it?

if a program made for a previous version of OSX is trying to use /usr/lib/libSystem.B.dylib it will fail, won't it?

Oooh, a double negative question, how to answer? (-:

The answer here is the majority of things Just Work™:

  • If your old program imports the library directly — using a LC_LOAD_DYLIB load command, visible in otool with the -L option — the dynamic linker knows that this library is in the shared cache and will fulfil the load command from there.

  • If your old program imports the library at runtime — using dlopen or some such API — the dynamic linker will similarly fulfil the request from the shared cache.

The only way to get into trouble here is to do something weird. For example, if your dlopen code preflights the request by checking to see if a file exists at that path, the preflight will fail.

All-in-all, almost a year into the macOS 11 roll-out, I’m impressed at how little has been broken by this change.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  • When I look into the liblas.tbd file in the /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/lib folder I see the line install-name: '/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib' but this library is nowhere to be found on my system.

    ls /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib ls: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib: No such file or directory
  • Hi Quinn,

    may I kindly ask your opinion on these two threads regarding the missing /usr/lib/libSystem.B.dylib?

    (https://github.com/iains/gcc-darwin-arm64/issues/88)

    and

    (https://trac.macports.org/ticket/64896)

    I hope they are not off-topic here (and please excuse me if they are, but this post I've found was the closest to the issue I'm facing), but the conclusion there seems to be that the error

    dyld[4691]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed dyld[4691]: Library not loaded: /usr/lib/libSystem.B.dylib   Referenced from: /Volumes/xHD/ndsha/Shared/Test/Rpath/static_complex   Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file) is misleading for that specific case, and possibly has to do with an Apple/ld64 thing, not related to gcc or fortran or macports or ... any such thing

    Your opinion would be much appreciated

Add a Comment

but this library is nowhere to be found on my system.

Right. That’s exactly what my post above explains.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

may I kindly ask your opinion on these two threads

I took a quick look and those seem like very different issues than the issue covered by this thread. I recommend that you start a new thread for that. Tag it with Linker so that I see it. Feel free to link back here if there’s something specific on this thread you think is relevant to the discussion.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hello Quinn,

I'm reviving this slightly old thread. Your explanation of the dynamic library cache and its history in this thread has been very useful. I have a few related questions. You note:

So in recent Xcode releases we’ve replaced the libraries in your SDKs with stub libraries [2]. That is, there’s no longer a libpcap.dylib library in the SDK, but rather a much smaller libpcap.tbd stub library [3]. It’s a text file (.tbd stands for text based description) so feel free to open it up and take a look.

  1. Consider one such .tbd file that's shipped in macOS M1 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/AppKit.framework/AppKit.tbd. This file doesn't contain the dynamic libraries that this library depends on. Is that intentionally left out? Tools like otool would be able to use that info to list the dependencies of the library if that info was present, isn't it?

  2. You noted:

Given the above reality, the libraries in /usr/lib are no longer needed: ...

This is fine if you’re using Xcode, or Apple’s command-line tools, because they know how to look inside an SDK for headers and stub libraries.

Is otool considered one of Apple's command line tools? The output of man otool does seem to suggest that it is one. If so, then do you think enhancing otool to seamlessly (or via a new option) look into the dynamic library cache if a path specified to that tool doesn't exist on the filesystem would be a valid enhancement?

I ask these questions in context of https://developer.apple.com/forums/thread/705281 where I ended up having to use external tools to extract the cache to get hold of the information I was after.

Finally, is there a doc/schema which explains the structure/contents of .tbd files. For example, I see that each of these file contents end with ... which at first looked odd to me and made me check if the rest of the contents in that file was missing.

Is that intentionally left out?

Yes. The build-time linker doesn’t care about those dependencies.

Moreover, this can change over time so, if it were present it may well be out of date. The SDK is fundamentally a build-time resource that’s independent of the currently installed OS. This is what allows you to build for iOS on macOS, or build for macOS 12 on macOS 11.

Is otool considered one of Apple's command line tools?

Well, that’s a matter of definition. That quote was referring to the tools used at build time — things like clang and ld — and those are fully SDK aware. In contrast, otool is a tool for inspecting libraries and, as you’ve noticed, it has no smarts to deal with the dynamic linker shared cache.

I ask these questions in context of

Oh, thanks for that context. The good news here is that we do have a way to achieve that goal, namely dyld_info. For example:

% sw_vers
ProductName:	macOS
ProductVersion:	12.3.1
BuildVersion:	21E258
% ls /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit                   
ls: …/AppKit: No such file or directory
% dyld_info -dependents /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit [x86_64h]:
    -dependents:
        attributes     load path
        re-export      /System/Library/PrivateFrameworks/UIFoundation.fram…
        upward         /System/Library/PrivateFrameworks/RemoteViewService…
        upward         /System/Library/PrivateFrameworks/XCTTargetBootstra…
                       /System/Library/PrivateFrameworks/InternationalSupp…
        weak_import    /System/Library/PrivateFrameworks/UserActivity.fram…
                       /usr/lib/libspindump.dylib
                       /System/Library/Frameworks/UniformTypeIdentifiers.f…
…

If you find yourself in a situation where you need to look inside the dynamic linker shared cache for some reason, and that’s not covered by an existing tool, I encourage to file an enhancement request for better tooling. Make sure to describe your requirements rather than focus on one specific tool. As the example above shows, the tool to enhance isn’t always obvious.

Finally, is there a doc/schema which explains the structure/contents of .tbd files.

Not that I’m aware of. I agree there there should be and I’d appreciate you filing a bug requesting that. Please post your bug number, just for the record.

Having said that, the format is relatively straightforward and so you can usually work out what you need to know just by rummaging around in same examples.

There are a variety of command-line tools to work with these files:

% apropos tapi
tapi(1)            - Text-based Stubs Tool
tapi-analyze(1)    - Analyze that a binary can be launched without missing frameworks and interfaces
tapi-archive(1)    - Merge or thin text-based stub files
tapi-installapi(1) - Create a text-based stub file by scanning the header files
tapi-reexport(1)   - Create a linker reexport file by scanning the header files
tapi-stubify(1)    - Create a text-based stub file from a library

In this context tapi stands for text-based API.

For example, I see that each of these file contents end with ... which at first looked odd to me

Indeed. That’s a standard part of YAML.

And yeah, I only know about that because I looked it up a few minutes ago (-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I'm hitting a brick wall within Python 3.9 on this issue. the cache access is at Python3.9 /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/types/macholib/dyld.py on line 11 but doesn't do much.

What I see is that the pattern 'X.framework/X' is necessary as a file - otherwise nothing works.

  from ctypes import util
  from ctypes.macholib.dyld import framework_find

# good
  pa = util.find_library('Python')
  pb = framework_find('Python')
  pc = framework_find('Python.framework')

# not good.
sa = util.find_library('System')
sb = framework_find('System')
sc = framework_find('System.framework')

ga = util.find_library('OpenGL')
gb = framework_find('OpenGL')
gc = framework_find('OpenGL.framework')

I’m not really set up to debug Python internals but my reading of that code in the latest Python 3.9 is (python-3.9.11-macos11.pkg) is that it’s has a specific affordance for based on _dyld_shared_cache_contains_path:

if os.path.isfile(path):
    return path
try:
    if _dyld_shared_cache_contains_path(path):
        return path
except NotImplementedError:
    pass

Do you see that in your version?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Oh, thanks for that context. The good news here is that we do have a way to achieve that goal, namely dyld_info. For example ...

Excellent. Thank you very much. This is exactly what I was after. The only reason I used otool was because of my lack of knowledge about dynamic libraries and tools in macOS.

If you find yourself in a situation where you need to look inside the dynamic linker shared cache for some reason, and that’s not covered by an existing tool, I encourage to file an enhancement request for better tooling. Make sure to describe your requirements rather than focus on one specific tool.

Will certainly do going forward.

Finally, is there a doc/schema which explains the structure/contents of .tbd files.

Not that I’m aware of. I agree there there should be and I’d appreciate you filing a bug requesting that. Please post your bug number, just for the record.

I will do that shortly and post the id here.

Slightly off-topic, but just wanted to share some feedback about the "Feedback assistance program" (the one for filing bugs) itself - I find that, that tool/process feels like "/dev/null" where you file an issue and see no updates to it at all. I don't just mean comments/responses, but even basic triaging updates like some human responding saying that it's being looked into. This observation is based on more than one issue that I have filed. The other thing with that tool is, you can't see other open issues - unlike other issue trackers where this is a common feature. So it just feels like an area where the person who has filed the issue is just talking to themselves. I'm really glad that at least the forums here are much more responsive with knowledgeable users.

For example, I see that each of these file contents end with ... which at first looked odd to me

Indeed. That’s a standard part of YAML.

And yeah, I only know about that because I looked it up a few minutes ago (-:

Thank you for that detail. I didn't even know it was a YAML format file :)