This aim of the post is to raise some questions and hopefully start a discussion around the use of test plans with unique configurations.
My team and I have set up a range of test plans with the aim of launching the application in a new state. Such states can have a dramatic effect on the UI and configuration of the launched application. However, upon doing we so have run into a range of issues concerning tests failures when switching configurations.
Upon further investigation it has been determined that the simulators are often launched with the prior configurations upon the completion of a given UI test. As the state of the application has not changed, the tests that rely on the correct version of the application to run, fail.
More specifically, when running the individual tests within Xcode, the test pass. However, when ran as part of the test plan, failures can be observed very consistently.
I am happy to provide any supporting documents to continue this discussion.
The questions we have here are:
- Is this a problem that anyone has encountered previously?
- Is this an intended limitation to the use of test plans?
- Can anything be done to mitigate this?
We believe the use of test plans is appropriate here and would like to adopt them if this problem could be resolved, thus, reducing code duplication for each state and streamlining our test setup.