How does a C program compiled from the command line link against a library in the Big Sur dynamic library cache?

I see Big Sur has put all the system provided libraries, including libc, into a built in dynamic linker cache. For some reason this cache is not visible to ld. This means using ld with -lc does not work as it used to.
There is a work around where you can use -L to add a path to the copy of the libraries that come with XCode, but does that mean anyone who wants to use the compiled program will need to have XCode installed?
Even if XCode is not required, I was wondering; what is the official Apple approved method for a C program to link against a library in the new dynamic linker cache?
It seems to me that ld with -lc should do this automatically. But if the official answer is to use dlopen, then how does a command line C application call dlopen unless it can link against the system library to gain access to dlopen?
What am I not understanding about the new way of doing things?

Answered by DTS Engineer in 653168022
Most folks compile C programs using the compiler driver. For example:

Code Block
% sw_vers
ProductName: macOS
ProductVersion: 11.0.1
BuildVersion: 20B29
% xcode-select -p
/Applications/Xcode.app/Contents/Developer
% cat test.c
#include <unistd.h>
const unsigned char pmessagebuf[13] = "hello world\n";
int main (int argc, char* argv[])
{
write(STDOUT_FILENO, (void*)pmessagebuf, 12);
return(0);
}
% cc test.c -o test
% ./test
hello world


If you want to compile each file separately, you can use the compiler driver for that too:

Code Block
% rm test
% cc -c test.c
% cc test.o -o test
% ./test
hello world


If you want to explicitly run the linker, you’ll need to tell it what SDK to use. A good way to work out the exact syntax is to run the compiler driver with -v:

Code Block
% cc test.o -v -o test
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld"
-demangle
-lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib
-dynamic
-arch x86_64
-platform_version macos 11.0.0 11.0
-syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
-o test
-L/usr/local/lib
test.o
-lSystem
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a


The ‘secret sauce’ here is -syslibroot.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
macOS does not come with with a built-in compiler. If you want to compile C code, you must either install Xcode or install the command-line tools package. In both cases you get a macOS SDK, and that SDK includes the stub libraries (.tbd) needed by the linker.

Linking against the runtime libraries built in to the OS hasn’t been best practice for a long time (ever since we introduced SDK support to Xcode) and in macOS 11 it’s finally broken.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
Thank you for the reply Quinn. I think I understand your answer.

I am using XCode and the command line tools and linking against the .tbd stub libraries in the SDK. But to do it I had to specify the path to the stub files since the path to the stub files does not lie along the standard shared library search path automatically.

What I'm worried about, and it's probably nothing to worry about, is that someone who uses the compiled program and does not have XCode installed, will have problems linking to the libraries. I'm guessing when ld builds an executable and you use -lc to tell ld to link against the c standard library, the path to the library is not built into the executable. Instead, will the run time linker then use the shared cache for the libraries?

I can't test this theory easily since I only have one machine and it has XCode with the command line tools installed. Uninstalling and reinstalling XCode and tools to test it is doable, but I still wouldn't know if the uninstall process completely removed everything... so I figured I'd ask you instead.

I am using XCode and the command line tools and linking against the .tbd stub libraries in the SDK. But to do it I had to specify the path to the stub files

Why?

since the path to the stub files does not lie along the standard shared library search path automatically.

What happens when you try to compile without specifying that path? What error message do you get? What does your compile command look like?

What I'm worried about, and it's probably nothing to worry about

Don't you think that if there was a problem, someone would have noticed it by now?

I'm guessing when ld builds an executable and you use -lc to tell ld to link against the c standard library, the path to the library is not built into the executable. Instead, will the run time linker then use the shared cache for the libraries?

The path to the library is is built into the executable. You can inspect that with "otool -L". The fact that the operating system uses the shared cache is an implementation detail.

I can't test this theory easily since I only have one machine and it has XCode with the command line tools installed.

Don't worry. About 50 million other people have thoroughly tested it by now. If you do have concerns about how your app would run in a "factory fresh" computer, you can install a virtual machine like Parallels and test it there. I strongly recommend that. Many developers install dynamic libraries from various "ports" or "brew" tools, from crazy cross-platform development tools, and are completely helpless when their apps don't work properly anywhere except their development machines.



What happens when you try to compile without specifying that path? What error message do you get? What does your compile command look like?


Compiling this code
Code Block language c
#include <unistd.h>
const unsigned char pmessagebuf[13] = "hello world\n";
int main (int argc, char* argv[])
{
write(STDOUT_FILENO, (void*)pmessagebuf, 12); 
    return(0);   
}


using these commands:
cc -c ./testcprogram.cpp -mmacosx-version-min=10.11 -mstackrealign
ld -execute -macosxversionmin 10.11 -lSystem -o testcprogram testcprogram.o

Does not work. You get the error that the system library can not be found.
Instead you have to use this linker command:
ld -execute -macosxversionmin 10.11 -lSystem -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -o testcprogram testcprogram.o


Don't you think that if there was a problem, someone would have noticed it by now?

It is a known issue in Big Sur and is in the release notes. I only found out about the work around because a lot of others on the internet already figured it out and posted about it.

The path to the library is is built into the executable. You can inspect that with "otool -L". The fact that the operating system uses the shared cache is an implementation detail.

If that's true, then you can't use ld to link .o files that have calls to the system library for production software anymore because the libraries will not be able to be found when xcode and the command line tools are not installed.


Don't worry. About 50 million other people have thoroughly tested it by now. If you do have concerns about how your app would run in a "factory fresh" computer, you can install a virtual machine like Parallels and test it there. I strongly recommend that. Many developers install dynamic libraries from various "ports" or "brew" tools, from crazy cross-platform development tools, and are completely helpless when their apps don't work properly anywhere except their development machines.

As you can see, I'm not using a crazy cross-platform development tool to compile this code. I'm only using the XCode command line tools cc and ld. And yes, there is a problem. I was hoping to hear Apple has say. Like if this will be fixed in the next release, or if support for ld is dropped, or if somehow using -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib with ld will work for production software. But from what you are saying, it looks like -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib is not a good solution.











ld -execute -macosxversionmin 10.11 -lSystem -o testcprogram testcprogram.o

Does not work. You get the error that the system library can not be found. 

You are using ld directly to link an executable? Why? I've never seen anyone do that.

Instead you have to use this linker command:
ld -execute -macosxversionmin 10.11 -lSystem -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -o testcprogram testcprogram.o

OK. So? You are doing something totally unique and you found a workaround - for today at least.

It is a known issue in Big Sur and is in the release notes.

I haven't see anything in the release notes about using ld. Maybe you are referring to the dynamic library cache issue. That is only incidentally related to ld.

I only found out about the work around because a lot of others on the internet already figured it out and posted about it. 

Don't take advice from the internet. They don't know what they are talking about. Just use Xcode. If you absolutely must develop your apps just one level above using a hex editor, what's the difference if you have to specify the path?

If that's true, then you can't use ld to link .o files that have calls to the system library for production software anymore because the libraries will not be able to be found when xcode and the command line tools are not installed.

The ld tool is only installed as part of either Xcode or the command line tools. Ergo, Xcode or the command line tools would have to be already installed if you are using ld. But that's not the point. At best, you are talking about a small implementation detail on a low-level tool that virtually no one uses, and obviously few of those should be using it.

As you can see, I'm not using a crazy cross-platform development tool to compile this code. I'm only using the XCode command line tools cc and ld. And yes, there is a problem. I was hoping to hear Apple has say. Like if this will be fixed in the next release, or if support for ld is dropped, or if somehow using -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib with ld will work for production software. But from what you are saying, it looks like -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib is not a good solution.

I'll accept that you aren't doing anything cross-platform, but I won't go any further than that. I'm not sure what you are doing here, in more ways than one. Generally speaking, the command line tools are for building and running software on your own machine. They shouldn't be used for distributing software. That's what Xcode is for. I don't know if the command line tools would create universal binaries, for example.

But regardless, there isn't anything wrong with using "-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib", just as there isn't anything wrong with using "-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib". Neither path is going to show up in the executable. That's why I suggested using "otool -L" to see how it works. It is going to say "/usr/lib/libSystem.B.dylib" using "otool -L". And you can even change that if you want, but that wouldn't be appropriate in this case.

You are using ld directly to link an executable? Why? I've never seen anyone do that.

It's C 101.

OK. So? You are doing something totally unique and you found a workaround - for today at least.

I know I'm getting old, but hopefully they still teach this in school. We're talking about using ld at its most fundamental level.
ld is used to link things together, including the system library. What you are technically saying is Apple is dropping support for using ld to link production executables. I hope you don't work for Apple.


I haven't see anything in the release notes about using ld. Maybe you are referring to the dynamic library cache issue. That is only incidentally related to ld. 

Yes, I am referring to the dynamic library cache issue. This is what broke ld. The Apple work around for the dynamic library cache issue is to use dlopen. Unfortunately, if your executable has to link against the system library to gain access to dlopen, and an ld compiled executable can't 'see' the system library because it's in the cache, then you can't use ld to link production software anymore. If this is the case, then Apple should say it is dropping support for the command line tools and all code must be compiled using the XCode IDE.

However, the -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib may work... I just have to test it. It's just ridiculous to have to tell ld what the path is to the libraries. The default paths for the ld linker tool should include everything it needs for linking against the system library once you install the command line tools.

What I'm really curious is how the XCode IDE is linking production executables. They must not be using ld.









Accepted Answer
Most folks compile C programs using the compiler driver. For example:

Code Block
% sw_vers
ProductName: macOS
ProductVersion: 11.0.1
BuildVersion: 20B29
% xcode-select -p
/Applications/Xcode.app/Contents/Developer
% cat test.c
#include <unistd.h>
const unsigned char pmessagebuf[13] = "hello world\n";
int main (int argc, char* argv[])
{
write(STDOUT_FILENO, (void*)pmessagebuf, 12);
return(0);
}
% cc test.c -o test
% ./test
hello world


If you want to compile each file separately, you can use the compiler driver for that too:

Code Block
% rm test
% cc -c test.c
% cc test.o -o test
% ./test
hello world


If you want to explicitly run the linker, you’ll need to tell it what SDK to use. A good way to work out the exact syntax is to run the compiler driver with -v:

Code Block
% cc test.o -v -o test
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld"
-demangle
-lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib
-dynamic
-arch x86_64
-platform_version macos 11.0.0 11.0
-syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
-o test
-L/usr/local/lib
test.o
-lSystem
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a


The ‘secret sauce’ here is -syslibroot.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

It's C 101....I know I'm getting old, but hopefully they still teach this in school. We're talking about using ld at its most fundamental level. ld is used to link things together, including the system library. What you are technically saying is Apple is dropping support for using ld to link production executables. I hope you don't work for Apple.

Rest assured, I do not work for Apple. For what it's worth, ld has virtually nothing to do with the C language. I don't know if you are trying to insinuate that I'm too young to know about these things or what. I can also assure you that I pre-date both ld and C.

What I'm really curious is how the XCode IDE is linking production executables. They must not be using ld.

If you are ever curious about how Xcode does things, you can always compile your little example code in Xcode and look at Xcode's link phase. It will look very similar to this:
Code Block
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -target x86_64-apple-macos11.1 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -L/Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Products/Debug -F/Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Products/Debug -filelist /Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Intermediates.noindex/ldtest.build/Debug/ldtest.build/Objects-normal/x86_64/ldtest.LinkFileList -Xlinker -object_path_lto -Xlinker /Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Intermediates.noindex/ldtest.build/Debug/ldtest.build/Objects-normal/x86_64/ldtest_lto.o -Xlinker -export_dynamic -Xlinker -no_deduplicate -Xlinker -no_adhoc_codesign -Xlinker -dependency_info -Xlinker /Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Intermediates.noindex/ldtest.build/Debug/ldtest.build/Objects-normal/x86_64/ldtest_dependency_info.dat -o /Users/jdaniel/Library/Developer/Xcode/DerivedData/ldtest-bvqaqdgevmzgmaedsishnfqvgzdl/Build/Products/Debug/ldtest


You don't have to type all of that yourself. You can just do:
clang -o testcprogram testcprogram.c

There are always other ways to do it. But once you start doing funky stuff, you risk getting stuck when something changes and you are the only one in the world doing it that way. In that case, if Apple breaks something, it's your problem. If you do things in a more traditional way and Apple breaks something, it's Apple's problem.




If you want to explicitly run the linker, you’ll need to tell it what SDK to use. A good way to work out the exact syntax is to run the compiler driver with -v:

Code Block
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -ltolibrary /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch x8664 -platformversion macos 11.0.0 11.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -o test -L/usr/local/lib test.o -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclangrt.osx.a

The ‘secret sauce’ here is -syslibroot.

You can inspect that with "otool -L".


Thanks Quinn and etresoft. I was able to link the clang generated helloworld.o file using this command:

ld -demangle -dynamic -arch x8664 -platformversion macos 11.0.0 11.1 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o testcprogram -L/usr/local/lib testcprogram.o -lSystem

I also verified with otool that the only library path for dyld and the system library in the executable is /usr/lib

I was also able to link a helloworld .o file generated using an assembler I wrote.

I noticed that -execute is no longer required for ld, and that -platform_version has replaced -mmacosx-version-min. I also couldn't find any documentation on -syslibroot. I'm guessing that the documentation for these changes will come sometime in the future.



I was also able to get this assembly language hello world to work:

Code Block
// Hello World for Big Sur
.text
_helloworldmsg:
.asciz "Hello World!\n"
.globl _myhelloworldstart
_myhelloworldstart:
    // syscall is deprecated.
    // Last I checked, if you use syscall, the assembler compiles an imported function call.
    //  If you hard code the syscall opcode, macos will not execute it and will give you an error
    //  so I am using imported function calls to access the system library functions.
    andq $0xfffffffffffffff0, %rsp // Force alignment. Without this exit will seg fault.
    movq $1, %rdi
    leaq _helloworldmsg(%rip), %rsi
    movq $13, %rdx
    call _write 
    xorq %rdi, %rdi // clear rdi to return success
    call _exit

using these commands:

as -mmacosx-version-min=10.11 ./helloworldbigsur.s -o helloworldbigsur.o

ld -e myhelloworldstart -nouuid -noehlabels -demangle -dynamic -arch x8664 -platformversion macos 11.0.0 11.1 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o helloworldbigsur -L/usr/local/lib helloworldbigsur.o -lSystem


How does a C program compiled from the command line link against a library in the Big Sur dynamic library cache?
 
 
Q