I have a Swift package with a test suite that contains some tests implemented with Swift Testing. Locally, they run quickly, but when I run them on Semaphore CI (https://semaphore.io), the first Swift Testing test to execute incurs a performance penalty.
I'm running the tests with xcodebuild on iOS Simulator:
xcodebuild test \
-scheme "Scheme" \
-workspace Workspace.xcworkspace \
-destination "platform=iOS Simulator,name=iPhone 16,OS=18.2"
The scheme is configured to use a test plan that has parallelization disabled.
Here's an excerpt from the output showing what I'm seeing:
Test Suite 'All tests' started at 2025-04-03 07:47:37.328.
◇ Test run started.
↳ Testing Library Version: 102 (arm64-apple-ios13.0-simulator)
◇ Iteration 1 started.
◇ Suite <redacted> started.
◇ Test foo() started.
✔ Test foo() passed after 23.063 seconds.
When foo() is not the first test it runs in under 100 ms.
The reason that I have parallelization disabled is that I was initially seeing all of the tests in this suite incur a performance hit. But now it's clear that there must be some startup cost.
Things I'm wondering:
What is this startup penalty?
Why don't I encounter it locally?
Why is it attributed to the first test? (this seems like a bug)
My wild guesses around 1 so far have been…
maybe some simulator clone is booting. I've tried to rule that out by disabling parallelization, but maybe there's still something there.
maybe swift testing is getting loaded lazily and there's some kind of dynamic linking cost
Thoughts on 2…
maybe there's some one-time penalty when using swift testing that I've already incurred locally but that has not yet been incurred in the CI image
Guidance welcome!
x-posted:
FB17102970 (Unexpected performance penalty attributed to first Swift Testing test)
https://forums.swift.org/t/first-swifttesting-test-always-slow/79066
1
0
185