Xcode 15 performance tests ignore baseline

I'm trying to write performance tests with Xcode 15, but it's not working: even though setting a baseline seems to work (baseline data files are created), Xcode seems to ignore that baseline altogether whenever the test is run. As a result, performance tests never fail, no matter how bad performance gets.

The source editor persistently displays a “No baseline average for Time” message.

Has anyone else encountered this? Figured out how to sidestep the issue?

Of note, the issue is easily reproduced: create a new (macOS in my case) app project with tests, add a dummy performance test; run the test once and save its baseline, and despite this subsequent tests will always succeed, even if you significantly slow the dummy test down.

(I've filed this as FB13330139)

Post not yet marked as solved Up vote post of Cykelero Down vote post of Cykelero
936 views
  • Yes, I also see this in Xcode 15.1 (15C65) running on 14.2.1.

  • Same result here with Xcode 15.1

  • Seeing the same issue in Xcode 15.3

Replies

I had the same issue. The baseline is now working on Xcode 15.2

The baseline is partially working for me on Xcode 15.2. It doesn't appear performance tests with the measure(metrics:block:) method are taking their baseline into account. Also Xcode is not suggesting setting а baseline for such a test. Tests with the measure(_:) method are working fine.

  • Intriguing—in my case, I only using measure(_:), which doesn't work.

Add a Comment

in addition to my previous comment