Xcode Server fails to produce installable product

I am getting this error when performing test action, using the same software versions.

I'm think I'm also in this boat. OS X 10.11.4 (Build 15E65), Server 5.1 (Build 15S5127), Xcode 7.3 (7D175).

I've done a lot of research on this issue and it all seems to point to the certificate needing permission before it can be used to codesign the application. Basically it stalls waiting for input from something before it will proceed.

That *sounds* like that could explain the problem, but how did you figure that out? Is there a log message I should look for to confirm that we're having the same issue?

Enviroment:

- Mac OS El Captain 10.11.4 (15E65)

- Xcode Version 7.3 (7D175)

- Mac OS Server Version 5.1 (15S5127)

I am struggling with a similiar issue and for me it boils down to two scenarios:

- sign the application with App Store Provisioning profile + Key - finishes without issues, no timeout

- sign the application with Add Hoc Provisioning profile + Key - times out on codesign

What actually times out in the end of the buildprocess is this command that creates an .ipa from .xcarchive:

xcodebuild -exportArchive -archivePath MyApp.xcarchive -exportPath ExportedProduct -exportOptionsPlist ExportOptions.plist -IDEPostProgressNotifications=YES -DVTAllowServerCertificates=YES -DVTSigningCertificateSourceLogLevel=3 -DVTSigningCertificateManagerLogLevel=3 -DTDKProvisioningProfileExtraSearchPaths=/Library/Developer/XcodeServer/ProvisioningProfiles -configuration Release

Content of the ExportOptions.plist is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-/
<plist version="1.0">
<dict>
  <key>method</key>
  <string>same-as-archive</string>
  <key>onDemandResourcesAssetPacksBaseURL</key>
  <string>http:/
  <key>thinning</key>
  <string>&lt;thin-for-all-variants&gt;</string>
</dict>
</plist>

Also I have given all the needed access for codesign to the private key located in the System keychain and went throough the web from end to end. Still, it fails.

Thing is that if you look carefully on the processes running on the Mac OS Server while your build is running then you can see that CI is working tirelessly with signing all your modules so it is not codesign key access issue its an issue of the fact that it is SLOW. Try it in terminal and see how the params for codesign change:

ps aux | grep _xcsbuildd | grep codesign

Command that is doing a codesign looks like this:

/usr/bin/codesign -vvv --force --sign 6BA79575D96E0826D4F7634FB36EBB93B0775378 --preserve-metadata=identifier,resource-rules --entitlements /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/entitlementsVnE /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone\ 5s/Payload/MyApp.app

In my case this command is executed loads of times, all the frameworks used are signed (not just MyApp.app) and then all the possible output products are generated. In my manual xcodebuild command tests I am able to get all the result .ipa files and all I need to do is wait for 50 minutes... yes...:

drwxr-xr-x  19 myusername  staff       646 Apr 19 11:25 .
drwxr-xr-x   5 myusername  staff       170 Apr 19 11:25 ..
-rw-r--r--   1 myusername  staff   7617154 Apr 19 11:24 MyApp-iPad (4th generation)-etc.ipa
-rw-r--r--   1 myusername  staff   6588133 Apr 19 11:24 MyApp-iPad 2-etc.ipa
-rw-r--r--   1 myusername  staff   7404436 Apr 19 11:24 MyApp-iPad Pro (12.9-inch).ipa
-rw-r--r--   1 myusername  staff   7404233 Apr 19 11:24 MyApp-iPad Pro (9.7-inch).ipa
-rw-r--r--   1 myusername  staff   7404021 Apr 19 11:24 MyApp-iPad mini 3-etc.ipa
-rw-r--r--   1 myusername  staff   7404118 Apr 19 11:24 MyApp-iPad mini 4-etc.ipa
-rw-r--r--   1 myusername  staff   7617136 Apr 19 11:24 MyApp-iPhone 4S.ipa
-rw-r--r--   1 myusername  staff   7617505 Apr 19 11:24 MyApp-iPhone 5-etc.ipa
-rw-r--r--   1 myusername  staff   7404396 Apr 19 11:24 MyApp-iPhone 5s.ipa
-rw-r--r--   1 myusername  staff   8617369 Apr 19 11:25 MyApp-iPhone 6 Plus.ipa
-rw-r--r--   1 myusername  staff   7404458 Apr 19 11:24 MyApp-iPhone 6.ipa
-rw-r--r--   1 myusername  staff   8617501 Apr 19 11:24 MyApp-iPhone 6s Plus.ipa
-rw-r--r--   1 myusername  staff   7404598 Apr 19 11:24 MyApp-iPhone 6s.ipa
-rw-r--r--   1 myusername  staff   7404615 Apr 19 11:24 MyApp-iPhone SE.ipa
-rw-r--r--   1 myusername  staff   7404305 Apr 19 11:24 MyApp-iPod touch (6th generation).ipa
-rw-r--r--   1 myusername  staff   7617300 Apr 19 11:24 MyApp-iPod touch.ipa
-rw-r--r--   1 myusername  staff  17117216 Apr 19 11:25 MyApp.ipa

So in my case the issue lies in the /user/bin/codesign command being utterly slow signing loads of files it needs to sign when Xcode Server is running an integration on a Bot. Here is an example execution done on two accounts. One account is the one that XCode Server is using called _xcsbuildd located in /var/_xcsbuildd (not /Users as you might expect) and the other is executed with a general user that has his stuff in /Users folder.

_xcsbuildd codesign:

bash-3.2$ time /usr/bin/codesign -vvv --force --sign 6BA79575D96E0826D4F7634FB36EBB93B0775378 --preserve-metadata=identifier,resource-rules --entitlements /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/entitlementsVnE /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone\ 5s/Payload/MyApp.app
Warning: usage of --preserve-metadata with option "resource-rules" (deprecated in Mac OS X >= 10.10)!
/var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone 5s/Payload/MyApp.app: replacing existing signature
/var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone 5s/Payload/MyApp.app: signed app bundle with Mach-O thin (arm64) [com.myapp]


real    0m7.261s
user    0m3.950s
sys     0m0.638s

myusername codesign:

bash-3.2$ time /usr/bin/codesign -vvv --force --sign 6BA79575D96E0826D4F7634FB36EBB93B0775378 --preserve-metadata=identifier,resource-rules --entitlements /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/entitlementsVnE /var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone\ 5s/Payload/MyApp.app
Warning: usage of --preserve-metadata with option "resource-rules" (deprecated in Mac OS X >= 10.10)!
/var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone 5s/Payload/MyApp.app: replacing existing signature
/var/folders/4r/4ll97s596wvfb6d39zt0hzhw000087/T/XcodeDistPipeline.Td3/Root-iPhone 5s/Payload/MyApp.app: signed app bundle with Mach-O thin (arm64) [com.myapp]


real    0m0.730s
user    0m0.304s
sys     0m0.042s

Same command, same input parameters and 10x the time for single codesign execution and as mentioned above, those codesign calls are executed maybe 50 times per single intergation so when you multiply that by 7 seconds per condesign you get a timeout on the CI. I can set the timout on the CI to be 2 hours for a build, but does it make sense? Huh. When you add the fact that Xcode server is unable to do concurrent builds you end up with a CI stuck for a long time.

To wrap up the issue is happening on the CI only building with Add Hoc provisioning profile, works fine with App Store provisioning profile and the culprit in my case is the really slow codesign command execution only on the _xcsbuildd user. How unix user account can be connected to a single codesign running time is a mystery for me.

I am still stuck with manual singing of the apps to test for my testers which is annoying and time consuming. Seriously thinking about moving to Jenkins and using some third party software that does its job better then Apple's which is a disgrace.

You can increase the timeout with xcscontrol. Go on you server and change it in a terminal.

Time is in seconds

$ sudo xcrun xcscontrol --configure-integration-timeout 900

It's working for me now.

Of course I can, but for me the timeout value would have to be an hour, I mentioned that in my post and what I also mentioned is that it doesn't make sense for a build to take one hour as it is usless for continuous integration.

Another option here is to disable thinned application variants for your project (and just export a universal application in your bot).

And how will I do that. I can't find an option to turn if off.

Wish I could upvote this... but how do you build a single universal app?

Xcode 8 beta 1 includes a fix for this issue, where integrations will now correctly report progress exporting thinned variants of your app (and no longer time out). The ability to disable app thinning on a per-bot basis will be in an upcoming Xcode 8 beta.

That is another huge improvement!

Still not sure why you guys had to reinvent the wheel - but at least we are getting to a state where all major issues are fixed!

So good work, keep it up!

In our case it was due to broken permissions. It usually happens after the Xcode Server or macOS update.

This can be fixed with

$ sudo xcrun xcscontrol --fix-permissions

note you can get the universal build by using thinning option of <none> or just not including the key/value for thinning

<key>thinning</key>

<string>&lt;none&gt;</string>

It's really interesting that you got all the output products, when I have thinning set to an iPod7,1 I get an iPad 4th gen out, and when I &lt;thin-for-all-variants&gt; I only get 4 types out, plus the universal type.