UI tests on Xcode Cloud became unstable when switched environment to Sonoma

My project has UI tests with total length of ~50 minutes. I'm using Xcode Cloud for builds delivery, UI tests execution is the part of the workflow. When I was using Monterey as tests environment everything was fine, but now it's no longer available so i had to switch to Sonoma. Since this i wasn't able to run anything but a couple of separate UI tests, when I run all of them I'm having 2 major issues. Locally on my machine I have the same Xcode/MacOS/simulator versions and everything works perfectly, issues only appear on Xcode Cloud. Here are they:

  1. There is a huge time gap between build-for-testing is finished and test-without-building is started. The gap is more than 1 hour so the sum of build time, ui tests time and the gap results in more than 2 hours and tests execution fails with the timeout. Funny is when I try to run the a single UI test or just run unit tests this gap only lasts for like 2-6 minutes, but when I add more tests the gap goes huge and fails everything. Also I guess my company is paying for those gaps and runs of tests when they timeout because of it even tho it's not our fault.
  2. Second issue is that some UI tests just freeze in the meddle of execution. This happens when UI tests is trying to interact with text fields on modal view controllers, sometimes it happens during the text field lookup, sometimes - after it takes focus and test is trying to type things into it. This is cannot be dodged anyhow and it is not related to some specific code - I've intentionaly rewritten the whole view controller from UIKit to SwiftUI , changed the way I present it and the location of presentation. Looks like just having a text field on a modal view is enough for test to freeze.

Have anyone experienced something like this? Found any solutions?

  • And yeah, forgot about 3rd one: Sometimes UI test cannot even pass the login flow, via ASWebAuthenticationSession, the login form just disappears but nothing happens. This happens like in 1 of 4 tests runs randomly, never happens locally

Add a Comment