UI tests on Xcode Cloud became unstable when switched environment to Sonoma

Question

Created Mar ’24

Replies 3

Boosts 2

Participants 3

My project has UI tests with total length of ~50 minutes. I'm using Xcode Cloud for builds delivery, UI tests execution is the part of the workflow. When I was using Monterey as tests environment everything was fine, but now it's no longer available so i had to switch to Sonoma. Since this i wasn't able to run anything but a couple of separate UI tests, when I run all of them I'm having 2 major issues. Locally on my machine I have the same Xcode/MacOS/simulator versions and everything works perfectly, issues only appear on Xcode Cloud. Here are they:

There is a huge time gap between build-for-testing is finished and test-without-building is started. The gap is more than 1 hour so the sum of build time, ui tests time and the gap results in more than 2 hours and tests execution fails with the timeout. Funny is when I try to run the a single UI test or just run unit tests this gap only lasts for like 2-6 minutes, but when I add more tests the gap goes huge and fails everything. Also I guess my company is paying for those gaps and runs of tests when they timeout because of it even tho it's not our fault.
Second issue is that some UI tests just freeze in the meddle of execution. This happens when UI tests is trying to interact with text fields on modal view controllers, sometimes it happens during the text field lookup, sometimes - after it takes focus and test is trying to type things into it. This is cannot be dodged anyhow and it is not related to some specific code - I've intentionaly rewritten the whole view controller from UIKit to SwiftUI , changed the way I present it and the location of presentation. Looks like just having a text field on a modal view is enough for test to freeze.

Have anyone experienced something like this? Found any solutions?

Boost

Answer 1

Igor_s OP

Mar ’24

And yeah, forgot about 3rd one: Sometimes UI test cannot even pass the login flow, via ASWebAuthenticationSession, the login form just disappears but nothing happens. This happens like in 1 of 4 tests runs randomly, never happens locally

0

Answer 2

someonelikeme OP

May ’24

I'm seeing similar problems. I think UI testing in Xcode cloud cannot be used (yet...).

I have simple test code like this:

XCTAssert(buttonBack.waitForExistence(timeout: 1.0))

... so it's supposed to wait for existence of that button for 1 second, and then fail the test if it cannot find the button, right? It works fine locally (of course), but it also works fine in Bitrise. But when running in Xcode Cloud, it fails there and I have this log:

Screenshot 2024-05-15 at 16.05.47.png

Just look at the timestamps to the right, it's looking for that button way longer than the specified time. and it even retries a few times before failing the test there.

0

Answer 3

badpirate_grindr OP

3w

We are experiencing this as well. Don't know if Sonoma is related as we've been on it for a while, but having tests time out on us (if they go over 2 hours depending on the "gap"). It's frustrating too because those that time out provide no logs at all for the testing-without-building portion of the workflow.

Here is an example of one that we got to pass today, it is a workflow with 3 test components that should run in parallel (UI, Snapshot and Unit), normally it's much quicker but has been timing out / taking a long time for the past couple of days. -- To get these times I downloaded artifacts for each of the test flows, and took the start and end times from xcodebuild-build-for-testing.log and xcodebuild-test-without-building.log

You can see that there is a 45 minute or more gap in between when the builds finish and when the tests begin. If you look at the total time for each individual test, it pretty closely aligns with Test Finish - Build start -- Our UI Tests "Said" they took 1h 35m, But if tests were to have started as soon as build finished, it would have only been 36m total.

My suspicion is that there is a shortage of test runners (which are often on a different machine than the builders), and so the testing is queued. If this is the case it shouldn't count against our 2h time limit (so they don't time out), and it shouldn't count against our usage.

1