False "xcodebuild archive" fail on bamboo/ Monterey/ m1 chip/ Xcode 13.2.1

This is so strange I'm at a loss. Background: I am a DevOps engineer validating that our iOS build scripts can run via Bamboo on new Mac mini M1 machines. Out of the box, these machines are running Monterey, while our in-use machines are all still on Big Sur.

Tldr; After modifying security permissions for our custom CI keychain so it's accessible by Xcode's CLI tools (specifically, the codesign and security utilities) the "xcodebuild archive" command we run fails with NO logged errors, causing bamboo to fail the build. However, the "failed" command produces a VALID xcarchive file. If I force the archive step to exit with a zero status, the subsequent "xcodebuild exportArchive" step successfully generates a valid .ipa that can be submitted to Testflight and run as expected. I've never seen anything like this.

Details: Initially, our builds were failing with codesigning errors related to the CLI tools interacting with our custom keychain (we use a custom keychain so Bamboo only has access to the codesigning assets it needs, this is best practice, at least up til now.) We could build on the new Mac mini with Xcode IDE just fine, and we could even build from the commandline on the new Mac mini. However, running the same commands via ssh to the new Mac mini from my local machine produced the same error bamboo was getting.

We were getting errors like this:

23-Mar-2022 13:35:58	Warning: unable to build chain to self-signed root for signer "Apple Development: Elizabeth Goldstein (XXXXXXX)"
23-Mar-2022 13:35:58	/Users/bamboo/agent-home/xml-data/build-dir/path/to/Build/Products/QA-iphoneos/Shared.framework: errSecInternalComponent
23-Mar-2022 13:35:58	Command CodeSign failed with a nonzero exit code

We were already unlocking the keychain as part of the build script, and the private key was set to "Allow all access" for outside applications (two common issues that might raise this error.)

Finally, I rebuilt the system keychain by first deleting then manually re-importing Apple's root signing certificates from terminal like this:

sudo security import ./AppleWWDRCA.cer -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild

sudo security import ./AppleWWDRCAG3.cer -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild

Then I deleted and re-imported my company's signing assets into our custom keychain like this:

security import ./appstore-cert.p12 -k /Users/bamboo/Library/Keychains/our-ci.keychain-db -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild

security import ./dev-cert.p12 -k /Users/bamboo/Library/Keychains/our-ci.keychain-db -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild

So now, there are zero errors in the log that I can find, but bamboo still fails when we run these commands:

/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA clean build -configuration QA -derivedDataPath output -sdk iphoneos OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode 
   

/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA -sdk iphoneos -configuration QA -derivedDataPath output archive -archivePath build.xcarchive OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode

If I change it to

/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA -sdk iphoneos -configuration QA -derivedDataPath output archive -archivePath build.xcarchive OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode || exit 0

The final exportArchive command generates a valid .ipa

Any help or ideas would be greatly appreciated, since of course that's not a viable workaround, nor is pinning our OS to Big Sur forever, and we need to scale up our CI system. Thank you for reading this far!

Accepted Reply

Thanks Matt, it took us over a month to get there but your reply pointed us in the right direction. For anyone struggling with this, it turns out NONE of the work in my original post was necessary, even though it did make the errors in our Bamboo xcodebuild logs go away.

THE ANSWER

  1. I had to change the bamboo launch process on our Mac mini build nodes from a daemon to an agent AND ALSO move the plist file into the Bamboo user context instead of the default System context, ex: /Users/USER_NAME/Library/LaunchAgents) instead of the system (/Library/LaunchAgents/)

  2. We were trying this with both brand new M1 and Intel nodes, and for the M1 node we also need to set up Xcode to open via Rosetta. This doesn't seem necessary on the Intel node.

I cried real tears over this so I hope it saves someone else the grief!

Replies

This is a tough one. In general I suspect your issues are being caused by this piece:

However, running the same commands via ssh to the new Mac mini from my local machine produced the same error bamboo was getting.

This is due to the user session not completely being initialized and having access to the Keychain and system utilities that a full user session normally would have.

You could try creating a per-user launch agent and executing the codesign commands from the agent to see if that get's you anywhere with the Keychain, but it's really recommended to run these commands from a full user session.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Thanks Matt, it took us over a month to get there but your reply pointed us in the right direction. For anyone struggling with this, it turns out NONE of the work in my original post was necessary, even though it did make the errors in our Bamboo xcodebuild logs go away.

THE ANSWER

  1. I had to change the bamboo launch process on our Mac mini build nodes from a daemon to an agent AND ALSO move the plist file into the Bamboo user context instead of the default System context, ex: /Users/USER_NAME/Library/LaunchAgents) instead of the system (/Library/LaunchAgents/)

  2. We were trying this with both brand new M1 and Intel nodes, and for the M1 node we also need to set up Xcode to open via Rosetta. This doesn't seem necessary on the Intel node.

I cried real tears over this so I hope it saves someone else the grief!

I hate to bring back the tears but this:

for the M1 node we also need to set up Xcode to open via Rosetta.

is a really bad idea in general. Running Xcode under Rosetta is almost never the right option and, even when it is, it’s a short term fix at best. Rosetta is not going to be around forever.

If you build your project normally, as a logged in using running Xcode in the GUI, does it work without Rosetta?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"