Xcode CI stuck on "uploading screenshots"

Our Xcode Server runs all of our integrations just fine, except for UI tests. Any UI test on any bot runs the tests (I can see them run via Screen Sharing), but then gets stuck on "Uploading 10 screenshots..." and never recovers. I have to reboot the server to unstick it, but then it gets stuck on the next UI test. These UI tests run fine via Xcode on the developer's machine, and even on Xcode Server running on another host. I haven't found anything suspicious in the logs, though I might not be looking at the right log files.


The Xcode Server is running OSX 10.11 + Xcode 8 and we have UI tests for both OSX and iOS which exhibit the same issue.


How can I debug this issue?

Same here! Trying to go through logs to find what's going on 😐


Edit 10/04/2016 @ 10:34pm (UTC):


Can't find any logs of such processing step failing.


First of all, even most sources online refer to

/Library/Developer/XcodeServer/Logs/Integrations/

as the location for Integration logs, I found them at:

/Users/<username>/Library/Caches/XCSBuilder/

maybe changed for Xcode 8 + Server 5.2. So take a look there, just in case.


Nevertheless, this didn't take me anywere in terms of what's going on with those screenshots (or whatever blocks the Bot):


I can see in xcodebuild.log that the build ends up satisfactorily Archiving and Exporting the product.

And buildService.log obtains the results, processess coverage. The last step logged is "Processing device".

No errors, but nothing else comes up.


Further, I can see the screenshots have been taken, and are available in xcodebuild_result.bundle.


Alternatively, in the more generic Xcode Server logs at

/Library/Developer/XcodeServer/Logs/Integrations/

I can see an error in xcscontrol.output.log that happens arround the same time the other logs stop:

It seems like it is trying to renew a certificate through

http://developerservices1.apple.com/services/<id>/clientCert/renewCert.action

This returns an error with code "3245", and message "The certificate provided is not in renewal period".


No idea why that would be happening at that point in the build process.


BTW, instead of restarting, you can unlock it by killing XCSIntegrationService.

But won't continue normally, just finishes with"The integration did not complete as expected due to an internal error (integration was cleaned up after a restart)."

I found no different outcomes from trying:

  • with different simulators,
  • from re-creating the bot,
  • from cleaning derived data and caches


Very odd. Wondering if I should open a radar.

We re-built the server VM from scratch 3 times now, hoping that it was some sort of configuration error, but that hasn't helped. As a workaround, we built out a Mac mini in my office which seems to be working better. Is your Xcode Server running on real hardware or a VMware instance?


I filed a bug report (radar 28716847). If you're having the same problem, I suggest you file a bug also.

I've also had this problem, but only on an older slower mac mini (Mid 2010 model, 8GB RAM). The same integrations run fine on a newer (late 2012) Mac Mini. Both mac minis have El Capitan installed.

Xcode CI stuck on "uploading screenshots"
 
 
Q