Is anyone seeing flaky results when using parameterized test with pairs of (input, result) data?
I have several (5) tests for a given method. I create a zip sequence by zip([in1, in2, in3, in4, in5],[out1, out2, out3, out4, out5])
Sometimes, the test only runs 4 of the tests and fails to report a failure even though I deliberately place data that should cause a failure.
Sometimes, even though I only select one test to run, the test explorer goes crazy into a loop and I have to clear test results to get it to stop. Following a suggestion, I disabled running tests in parallel.
Xcode 16.2 / OSX 14.7 (Sonoma) / Mac mini M2 Pro
I wrote a simple test
import Testing
func target(value: Int) -> Int {
value + 1
}
struct ParameterizedTestTests {
@Test(arguments: zip([1,2,3,4,5],[2,3,4,5,5]))
func test1(x: Int, r: Int) async throws {
#expect(target(value: x) == r)
}
}
Note that the last test case should fail.
I've run it numerous times, including shutting down Xcode and restarting and the test works, reporting the failure for the last test case.
I went back into my original code and recreated problem test (I had rewritten the test to not use parameters). It was still acting up, but then I noticed in a previous parameterized test that the test was expecting UInt16 parameters, but the parameters given were Int (i.e. no type specified, which Swift defaults to Int). When I changed them to UInt16, it cleared things up. However, unlike a normal function call which complains if the type of the arguments don't match the type of the parameters, parameterized tests don't report a mismatch.