Swift Test Parameterized Test

Is anyone seeing flaky results when using parameterized test with pairs of (input, result) data?

I have several (5) tests for a given method. I create a zip sequence by zip([in1, in2, in3, in4, in5],[out1, out2, out3, out4, out5])

Sometimes, the test only runs 4 of the tests and fails to report a failure even though I deliberately place data that should cause a failure.

Sometimes, even though I only select one test to run, the test explorer goes crazy into a loop and I have to clear test results to get it to stop. Following a suggestion, I disabled running tests in parallel.

Xcode 16.2 / OSX 14.7 (Sonoma) / Mac mini M2 Pro

Answered by MarkErbaugh in 825865022

I wrote a simple test

import Testing

func target(value: Int) -> Int {
    value + 1
}

struct ParameterizedTestTests {
    @Test(arguments: zip([1,2,3,4,5],[2,3,4,5,5]))
    func test1(x: Int, r: Int) async throws {
        #expect(target(value: x) == r)
    }
}

Note that the last test case should fail.

I've run it numerous times, including shutting down Xcode and restarting and the test works, reporting the failure for the last test case.

I went back into my original code and recreated problem test (I had rewritten the test to not use parameters). It was still acting up, but then I noticed in a previous parameterized test that the test was expecting UInt16 parameters, but the parameters given were Int (i.e. no type specified, which Swift defaults to Int). When I changed them to UInt16, it cleared things up. However, unlike a normal function call which complains if the type of the arguments don't match the type of the parameters, parameterized tests don't report a mismatch.

I’ve not seen this, and I have run similar tests, but that doesn’t mean a lot. There are way too many variables in flight to draw any solid conclusions.

I hate to sound like a broken record, but have you tried extracting this into a small test project. That’ll help in a couple of ways:

  • If the problem reproduces there, it’s great bug fodder.

  • If not, that’s useful info too, in that it tells you that there are bigger factors in play.

Also, if you’re primarily interested in hearing about the experiences of others, you might try posting over in the Swift Testing area of Swift Forums. You’ll likely find a large community over there.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I wrote a simple test

import Testing

func target(value: Int) -> Int {
    value + 1
}

struct ParameterizedTestTests {
    @Test(arguments: zip([1,2,3,4,5],[2,3,4,5,5]))
    func test1(x: Int, r: Int) async throws {
        #expect(target(value: x) == r)
    }
}

Note that the last test case should fail.

I've run it numerous times, including shutting down Xcode and restarting and the test works, reporting the failure for the last test case.

I went back into my original code and recreated problem test (I had rewritten the test to not use parameters). It was still acting up, but then I noticed in a previous parameterized test that the test was expecting UInt16 parameters, but the parameters given were Int (i.e. no type specified, which Swift defaults to Int). When I changed them to UInt16, it cleared things up. However, unlike a normal function call which complains if the type of the arguments don't match the type of the parameters, parameterized tests don't report a mismatch.

Thanks for all that extra info.

Written by MarkErbaugh in 825865022
However, unlike a normal function call which complains if the type of the arguments don't match the type of the parameters, parameterized tests don't report a mismatch.

Right. Welcome to macros |-:

However, using a value out of range should be flagged. For example, I tweaked your test like so:

import Testing

func target(value: Int) -> Int {
    value + 1
}

struct ParameterizedTestTests {
    @Test(arguments: zip([1,2,3,4,100_000],[2,3,4,5,100_001]))
    func test1(x: Int, r: Int16) async throws {
        #expect(target(value: x) == r)
    }
}

and got a Integer literal '100001' overflows when stored into 'Int16' error deep within the macro expansion.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Swift Test Parameterized Test
 
 
Q