Missing librairies in /usr/lib on Big Sur?

To solve a dependency tangle on an app, I’m trying to write a simple command line tool that would display all the dependencies of a given app or library, and output it in a format suitable for post-processing by graphviz. The idea here is to collect and lay out the output of the otool -L utility, recursively called on all the dependencies of the target app/lib.

Unfortunately, when I try otool -L with, say, otool itself, I get this:

Code Block shell
Dev > otool -L /usr/bin/otool
/usr/bin/otool:
/usr/lib/libxcselect.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)

Fine. But now:

Code Block shell
otool -L /usr/lib/libxcselect.dylib
/Library/Developer/CommandLineTools/usr/bin/objdump: error: '/usr/lib/libxcselect.dylib': No such file or directory

Oops. Indeed, /usr/lib seems mostly empty, and most of what lies inside are links on missing (I assume: invisible) libs.

So my question is: where are all the libs gone, and it is possible to bring them back to the surface?

Answers

Maybe review the WWDC videos and/or post in a beta forum.
All of these libraries have moved into the shared cache. For more details, search the macOS Big Sur 11 Beta 3 Release Notes for 62986286.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
previous link is broken

 macOS Big Sur 11 Beta 3 Release Notesfor 62986286.

any other info available?

THX

previous link is broken

This morphed into the main macOS Big Sur 11.0.1 Release Notes.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
I am running into this problem when trying to use gfortran (which should be a subset of gcc ... but in fact is not in /usr/bin

eskimo: I assume you are refering to this paragraph:

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic
linker cache of all system-provided libraries. As part of this change,
copies of dynamic libraries are no longer present on the filesystem.
Code that attempts to check for dynamic library presence by looking for a
file at a path or enumerating a directory will fail. Instead, check for
library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)


But not sure how to interpret it... where is this 'cache'???

Doing, for example a locate libSystem.dylib

Code Block
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/tvOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/watchOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/DriverKit/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/tvOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/watchOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libSystem.dylib
/System/Volumes/Update/mnt1/System/DriverKit/usr/lib/libSystem.dylib

But the directory /System/Volumes/Update/mnt1/ is empty....
And I assume the others don't apply to a Big Sur on an older (2.2 GHz Quad-Core Intel Core i7) mac

Bottom line, which directory should I include to find

I am running into this problem when trying to use gfortran

OK.

I assume you are refering to this paragraph

Yes.

which directory should I include to find

That kinda depends on the capabilities of your toolchain. To understand this I need to explain some backstory…

Way back with the original release of Mac OS X we didn’t support SDKs. When you installed the developer tools it would lay down headers at absolute locations on your current system — for example, the <pcap/pcap.h> header would be placed at /usr/include/pcap/pcap.h — and tools would look for headers at those absolute locations. The same thing would happen for libraries, except the linker would look in /usr/lib.

So far, so Unix.

However, this design did not meet Apple’s needs. The most obvious problem here is that you can’t have multiple SDKs installed. This is a hassle because you can’t, for example, build your production product against the production headers and libraries while also building your in-development product against the next beta headers and libraries.

Apple solved this by creating SDKs. An SDK includes all the headers and libraries needed to build for a specific version of the OS. For example, the macOS 11.1 SDK included in Xcode 12.4 includes all the headers and libraries needed to build for macOS 11.1. Additionally, our toolchains support the concept of a deployment target, which allows you to build with the macOS 11.1 SDK but still run on, say, macOS 10.15.

Apple also used this SDK feature to support multiple platforms. Xcode now contains a macOS SDK, an iOS SDK, and so on.

One problem with these SDKs is that they’re rather large. Once significant contribution to that size was the libraries. Each SDK would include a copy of the libraries necessary to link for that platfrom. However, this is nonsense when you think about it. There’s no point shipping an iOS library on macOS because it’s just a bunch of code that can’t ever execute [1]. The only thing that the linker needs from these libraries is a list of symbols. So in recent Xcode releases we’ve replaced the libraries in your SDKs with stub libraries [2]. That is, there’s no longer a libpcap.dylib library in the SDK, but rather a much smaller libpcap.tbd stub library [3]. It’s a text file (.tbd stands for text based description) so feel free to open it up and take a look.

Given the above reality, the libraries in /usr/lib are no longer needed:
  • Developer tools should be looking at the stub libraries in the appropriate SDK.

  • The runtime doesn’t use these libraries because they’ve been rolled into the dynamic linker shared cache [4].

And that’s why they were removed.



This is fine if you’re using Xcode, or Apple’s command-line tools, because they know how to look inside an SDK for headers and stub libraries. If you’re using third-party tools then you need to consult the support resources for those tools to find out how they’ve adjusted to this new reality. In your case the critical point is the linker. If your third-party tools use the Apple linker then you should just be able to point the tools at the usr/include directory within the appropriate SDK. Our linker will automatically use any stub libraries that it finds.

For example, if you have Xcode installed in the normal place and you’re targeting macOS then the correct directory is:

Code Block
/Applications/
Xcode.app/
Contents/
Developer/
Platforms/
MacOSX.platform/
Developer/
SDKs/
MacOSX.sdk/
usr/
lib


OTOH, if your third-party tools are using their own linker then you’ll have to determine whether that linker has been updated to use stub libraries. I can’t help you with that, alas.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, nowadays we allow users to run iOS apps on Apple silicon but that’s besides the point (-:

[2] This is a term, and a concept, that originated on traditional Mac OS.

[3] Technically, both of these files are symlinks to the corresponding libpcap.A.*** files.

[4] This term is slightly off. Historically this was actually a cache that the system would build from its working set of libraries and frameworks. On macOS 11 this is no longer a cache because the original files are no longer present on the system. Rather, this shared cache is authored by Apple and distributed in toto via the software update mechanism.