I have a Swift package with a test suite that contains some tests implemented with Swift Testing. Locally, they run quickly, but when I run them on Semaphore CI (https://semaphore.io), the first Swift Testing test to execute incurs a performance penalty.
I'm running the tests with xcodebuild on iOS Simulator:
xcodebuild test \ -scheme "Scheme" \ -workspace Workspace.xcworkspace \ -destination "platform=iOS Simulator,name=iPhone 16,OS=18.2"
The scheme is configured to use a test plan that has parallelization disabled.
Here's an excerpt from the output showing what I'm seeing:
Test Suite 'All tests' started at 2025-04-03 07:47:37.328. ◇ Test run started. ↳ Testing Library Version: 102 (arm64-apple-ios13.0-simulator) ◇ Iteration 1 started. ◇ Suite <redacted> started. ◇ Test foo() started. ✔ Test foo() passed after 23.063 seconds.
When foo()
is not the first test it runs in under 100 ms.
The reason that I have parallelization disabled is that I was initially seeing all of the tests in this suite incur a performance hit. But now it's clear that there must be some startup cost.
Things I'm wondering:
- What is this startup penalty?
- Why don't I encounter it locally?
- Why is it attributed to the first test? (this seems like a bug)
My wild guesses around 1 so far have been…
- maybe some simulator clone is booting. I've tried to rule that out by disabling parallelization, but maybe there's still something there.
- maybe swift testing is getting loaded lazily and there's some kind of dynamic linking cost
Thoughts on 2…
- maybe there's some one-time penalty when using swift testing that I've already incurred locally but that has not yet been incurred in the CI image
Guidance welcome!
x-posted:
- FB17102970 (Unexpected performance penalty attributed to first Swift Testing test)
- https://forums.swift.org/t/first-swifttesting-test-always-slow/79066
It looks like you’re connected with the right folks via your Swift Forums thread.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"