Swipe issue UITesting + fastlane

We have a suite of UI tests managed by QA.

They are developed on his machine (M1 MacBook Pro) and when they all pass he checks in

We use fastlane and an M1 Mac-mini as our build machine/pipeline.

When the tests are running in the pipeline we always get swipe failures such as

XCTAssertTrue failed - Failed to swipe so not on the correct page

XCTAssertTrue failed - Swipe not worked, delete not showing

And so on. What I find really strange is I can login to the build machine itself, open the project from the source directory and manually run all the UI Tests - they all pass.

I also ran the fastlane lane directly (ui_tests) from terminal to exactly mimic what the pipeline does and they all pass

As soon as I re-enable them in the full pipeline (fast file) we get the swipe failures on the same machine and emulator!

The only difference I can see is I ran them directly / standalone versus part of the pipeline. Also, they run immediately after our unit tests

 unit_tests
 ui_tests

Could this potentially affect things? From reading around I wondered about the M1 architecture potentially being the cause but they are written on an M1 and the build machine is the same architecture (and as above, in isolation they work)

Any ideas appreciated!