Testing in Xcode

Good morning, and welcome to Testing in Xcode. My name is Ana Calinov and I'll be presenting along with my colleagues, Stuart Montgomery and Ethan Vaughan.

In today's session, we'll start with an introduction to testing in XCode with XCTest.

Then, Stuart will tell you about the test plans feature. Finally, Ethan will show you applications of XCTest with continuous integration.

Let's get started with XCTest. XCTest is the automation testing framework provided in Xcode with built-in support to help you set up and execute your tests. Testing is an important step of developing any project which can help you find bugs in your source code.

You can also use test to codify requirements, meaning you make test for expected behaviors for your app, and further work by you and your team can be qualified against these expectations. We're going to start with a summary of how you should consider planning out the automation test suite of any project.

The pyramid model approach to testing helps to strike a balance between thoroughness, quality and execution speed.

Unit tests are the foundation of our pyramid. Unit tests help verify a single piece of code, generally a function.

This is done by inputting variables to the function and checking that they return the expected output.

Unit tests are short, simple, and run very quickly.

This is the foundation of all of our testing so you want to write many unit tests to cover all your functions. Next, we have integration tests.

Integration tests are used to validate a larger section of your code.

These tests should target discrete subsystems or clusters of classes to make sure that different components behave correctly together. Integration tests sit on top of unit tests, because you want to make sure that the individual functions behave correctly before testing this larger piece of your code.

You generally also won't need as many integration tests as unit test.

They may take slightly longer to run but they do test more of your app at once. Lastly, user interface or UI tests observe the user-phasing behavior of your app.

This makes sure that your app truly does what you expect it to.

UI tests take the longest to run but they're vital to demonstrating that everything behaves correctly.

UI tests also require more maintenance because your app's UI may change more frequently.

The full pyramid of tests can therefore help you balance between these three different test types and ensures your test suite gives you the coverage that you need.

So we just went over how to balance your test suite, so now let's switch to the tools provided by XCTest to help you implement it. Unit tests in XCTest are all of your tests targeting your source code. This includes both your standard unit test and also your integration tests.

UI tests execute on top of your app's UI to provide end-to-end qualification of your app.

UI tests are also black box test because they won't rely on any knowledge of the functions or classes actually supporting your app.

UI tests will be able to make sure that everything behaves correctly at the very end.

Lastly, performance tests run multiple times over a given test to look at the average timing, memory usage or other metric given to it to make sure you don't introduce regressions in these areas. Today, we'll be focusing on unit and UI tests.

The easiest way to get started with testing in your Xcode project is to choose to include both unit and UI tests when starting a new project. In our brand new project, you can see the unit tests targeting class and the UI tests targeting class automatically created and displayed in your project navigator. You are also provided with the template to start writing each of these test classes and the test cases within them.

Now let's take a closer look at a test class that uses XCTest.

The class imports the XCTest framework along with the target to be tested.

The class itself is a subclass of XCTest case which allows its methods to be used by Xcode to execute tests. Each method we want to use as a test case must start with the word test and then hopefully be named something that indicates what it will do. You'll also see a test diamond appear to the left of the test to show that Xcode can execute it. Inside of the test, assertion APIs are used to evaluate and validate your source code. In this case, XCTAssertEqual will compare the first two values given to it and make sure that they're equal. If they differ, the test will fail instead.

Now, once we run this test, we hope that it passes and the test diamond turns green with the checkmark inside.

We know this isn't always the case though.

If the test fails, the test diamond turns red with an x and the relevant line will highlight.

We also get an error message that shows what went wrong in the test.

The string we pass to XCTAssertEqual also shows up now to give us more information to debug the issue. In this case, we may have an off by one error in our source code or you may have initialized to an unexpected value. We can go back to the source code, fix this issue, and run our test again until it passes. Each test class template will also include a setUp and a tearDown method. These blocks serve as a way for you to do any work you need to around your test in order to keep your tests specific to their purpose. SetUp is called before each of your test cases executes. In UI test, this helps ensure that your app is launched before you try interacting with it.

XCTest then runs your test method and afterwards tearDown can be used to clean up any changes you've made to the data or the global state of your app, to make sure your test leaves nothing behind that could impact subsequent tests.

Now, let's go into a demo to see how to write unit and UI tests for your app. So, we've been working on a travel app. This shows different destinations around the world and can help you plan a vacation.

I'd like to add a new feature to my app that shows how far away each of these destinations is from our current location in San Jose.

For this, I've written a new class called DistanceCalculator.

This class defines a city struct that contains a string with the name of the city and a tuple for its coordinates. I currently have my list of cities stored in a dictionary. I'm planning on moving these two database later though.

I have a function called city that will return an optional city struct type to be able to search my dictionary for the cities. The main function I'm planning on using from this class is distanceInMiles.

This takes in two strings with city names and returns the distance between them as a double. If either of the cities can't be found in our dictionary, an error will be thrown instead.

Lastly, I have another helper function also called distanceInMiles. This one takes in our city struct and returns the distance between them. This function uses the core location framework so that it does the heavy lifting for me.

Now, to start writing unit test for this class, I've created a new test class called DistanceCalculatorTests.

As I start writing tests, I want to see my source code so I'm going to open another editor below, and then navigate to my DistanceCalculator class.

The first test that I want to write is testing my city function. So I'm going to start writing a unit test and I can call it testCoordinatesOfSeattle. For each of my unit test classes, I'm going to choose a specific test case to make sure that my function works.

In this case, I'm going to input Seattle to my city function and make sure that its correct coordinates are returned.

The first thing I'll do is to find calculator by initializing my DistanceCalculator. Now, I can start writing and make a call to my function.

But one thing to remember is that city returns an optional city struct. So I want to make sure that the value I get is not nil.

To do this, I can use this API and try using XCTUnwrap to make sure that the value is valid.

Then I can call calculator for Seattle.

Now, I'll get an error message that appears, because it looks like there's an error that could be thrown that is not being handled.

If city returns a nil value, then I have to make sure that my test case also throws to properly show the issue.

Now, I have my city variable for the value return for Seattle and I can use my assertion APIs to make sure that both the latitude and the longitude are right.

Now, I'll run my test by clicking on the test diamond.

This will launch the app, run the test, and tell us what happened. It looks like our test succeeded. It's great. I can write another test now for my class and I'm going to focus on my distanceInMiles class.

My function is going to be called testSanFranciscoToNewYork to specifically look at what my functional return for the assistance.

Now the first thing I'm going to have to do here is to find another calculator by initializing DistanceCalculator.

But at this point, my code is getting repetitive because I have to do this at the beginning of each of my test cases.

So instead, I'm going to declare a class variable called calculator and make use of my setUp function to initialize it before each test begins.

Now I can start writing my test. I'll define distance in miles by trying to call distanceInMiles from my function from San Francisco to New York. And then I'll use my assertion APIs to make sure that the distance is correct. I'll run my test from the test diamond. And it looks like this test failed instead. I can look at the error message at the failure message in my issue navigator. This time our test runs and we get an error message with an actual test failure.

It looks like the value that we're expecting and comparing to has a lot more precision than the one we're actually looking for. In this case, I don't care about all this precision because I only want to show my user a whole number of miles between these cities.

So, I can add an accuracy argument instead of equal to one. This will allow XCTAssertEqual to have a leniency and allow the numbers to defer by one and still pass the test.

I'll write my test again to make sure it passes. Great. Now I've written two tests so far for my unit test class, but both of these tests take valid inputs to the functions and check that a valid input is returned. But there are some other test cases I want to check for my function. I want to make sure that errors are also handled correctly in my class.

So, the next function I'm going to write is going to be a negative test case. I don't expect Cupertino to be my database of cities because I'm only considering large cities, and Cupertino won't be one of them.

So, when I call Cupertino, I'll expect an error to be thrown to say that the city is unknown in our database. I can use XCTAssertThrowsError to do this. I can then use a closure to look at the exact errors that were thrown and compare them with XCTAssertEqual to make sure they're correct.

Now, I've written three tests and I want to run all of them at once to make sure that everything is working right.

So I can go to the test navigator and this will show me row by row, all of the different test targets, test classes, and test cases in my project.

I can select at each level to run all of the tests below it.

So, if click the play button next to DistanceCalculatorTest, it will run all three of these tests.

My Apple launch runs them and all the green checkmarks means they all passed. I can also command and click on different rows in order to select different values to run or different test categories to run if I don't want to run all of them at once.

By context clicking, I can choose to run them. If I do choose a subset to run and then want to rerun the same subset later after I fix some issues, I can also go to the Product menu, click on Perform Action and choose to rerun just the last test that was run.

Now that I've written some unit tests for my class, I want to implement it in the UI.

I'll do so by showing a distanceText and running my class. Now, under the city name, we have a distance that shows how far away these cities are from our current location. So, I've created a new UITest class called DiscoverUITest.

Our UITest class has set up populated with two things. Continue after failure is set to false. Once we failed a UI test, it usually means we've gone into a UI state that's unexpected. Chances are we won't be able to interact with anything further because we don't know what's actually on the screen. We also want to make sure that our app is launched before we try interacting with it. We can start writing our UI tests just as we were writing our unit test. I'm going to write testMilesToParis and the goal will be to open the app, swipe to the Paris icon and then make sure that the distance shown is correct.

The first thing I'm going to do is actually make sure that we're on the discover tab when our test starts. I can write UI test by looking for elements in the app's UI and interacting with them. In this case I'm going to look for a tab bar button called Discover. And then I'm going to tap on it to make sure we're on the right tab.

After every UI action we do, we want to verify that the correct screen is now displayed. So I'm going to then make sure that San Francisco is visible on the screen to make sure we're in the expected state. I'll use an XCTAssert statement to make sure that San Francisco, a static text isHittable. IsHittable will ensure both that the element exists and that is on the screen so we can interact with it. Now, the next thing I want to do is swipe left on this image to get to the Paris location. But I'm actually not sure how to interact with this image because they may have a custom label that I'm not sure how to define. So I can use the debugger to get more information about my app's UI. I'm going to set a breakpoint on line 26 and then run my test from the test diamond. The Apple launch click on Discover, make sure San Francisco is visible and then pause in the debugger.

From the debugger, we can get more information about the app.

We can print out the exact view hierarchy by using po app to get this information. Here we have a list of all of the UI elements in our app. This is a little overwhelming though so I'm going to further specify that I want to print out all of the images in my app only. Great. Now we have a list of all of these images and it looks like San Francisco is the one I want.

So I'm going to copy the string and I'll close the debugger and then define my sfImage to be an image with this identifier. Now, I can simply call on this query to swipe left.

Once I've done this, I want to confirm that Paris is actually visible on the screen so that no changes in the UI would interrupt this test.

So I can use XCTAssert to make sure that Paris is hittable.

Lastly, I want to make sure that this correct distance is displayed. So I'll use one more XCTAssert statement to make sure that 5586 miles, a static text, is visible.

Now we can run our test from the test diamond and we'll see the app launch and execute through each of these steps, once I remove my breakpoint.

So we'll see that the app launches, we press Discover, check for the text, swipe left and then make sure that all of our correct strings are displayed.

One more thing to consider with this test is that we're looking for a static text, which says exactly 5586 miles. You may want to make sure that you're mocking your location or simulating that you're in exactly in San Jose whenever you run this test because otherwise the distance will vary based on your actual location.

And now let's go back to the slide to talk about test organization in your project.

So when starting to write your test class, you'll start out with two test targets, generally, one for unit tests and one for UI tests.

Unit test and UI test must be separated by type because of the differences in how they execute on your app.

Each of these test classes-- test targets will contain your test classes.

Your test targets can have as many test classes as necessary to test your app.

Your test classes then contain each of your test cases.

Together, your unit test target and UI test target can fully test your app.

But there are some cases where you may need more test targets. As your project becomes more complicated you may want a new-- to create a new framework. This framework should have its own unit test target to contain its test.

Additionally, Swift packages that you create and write tests for in Xcode already define test targets. These test targets behave the same way in Xcode as any other unit test target. Finally, once you have a full test rewritten for each of your test targets you may wonder how well your test actually cover all of your source code. For this you can use Code Coverage. Code Coverage is a feature in Xcode in XCTest that will measure and visualize the number of times each line of your source code was executed during your test run.

After enabling the feature and running your test, you can go to the report navigator and select the coverage data for your test run.

There, you'll see a list of each of your test targets and classes along with the percentage that shows how much of your source code in that section was actually executed. You can select each test class to switch directly to the file.

When showing coverage data in a source file, the source editor gutter on the right will show a number indicating how many times that line was executed when you ran your tests.

You can also hover over the numbers to see information directly in your source editor.

Lines that were executed will turn green.

Sections that were not hit during your test will highlight red.

You can also see complex information showing individual code paths that were not hit during your test, such as conditionals that were never selected. The code coverage tool overall gives you more information about your test and can help you identify areas you may want to write more tests for. When committing new work to repository including your test along with your code ensures that everything is quality controlled and checking your code coverage ensures that you don't miss anything.

If you feel like your tests don't fully cover your changes, it may be a good time to go back and write more tests. You'll get the most value out of your tests by writing them early. By having a test suite that goes along with your source code, you can ensure that each new function you write is reliable and behaves as expected. And remember, testing is an ongoing process and vital to the health of your app. Now, I'd like to invite Stuart to the stage to tell you more about how you can get the most out of your tests in Xcode. Thanks, Ana. So now that you've learned the basics of testing in Xcode, I'd like to talk about a new feature in Xcode 11 called Test Plans, which helps you get the most out of your tests. So whether you're just getting started with writing test or if your project already has a large and robust test suite, there's one piece of advice we'd like to share to help you get the most out of your tests. We actually recommend that you run them more than once in different ways.

Now even if you don't modify your tests at all, if you leverage more of Xcode's build in testing options and its advance capabilities, you can get a lot more out of your tests and catch more bugs. Let's walk through an example to explain this. So, imagine that the app we've been working on is localized into several different languages.

And now that we've learned how, we've written several UI tests for our app as well.

Now, once we've done this, our test will most likely succeed when we run them in our development language, which in this case is English.

But imagine that one day we discover a bug that only reproduces in certain languages where a localization string is missing for that language. It's using a placeholder string instead which breaks the UI layout.

Now, once we're aware of this bug, we can adjust Xcode settings to manually run our UI tests in that language. And if we do that we might see a UI test failure. Of course if we don't, that's a great opportunity to write a new test that reproduces the issue and fails until we fix the bug.

But once we fix that issue and we have a test covering it, ideally we should always run our tests in this language in addition to our development language to make sure it doesn't break again. So that's just one example but there are other cases too of bugs that might only be caught if you run your tests more than once in a different way each time.

For example, you might choose to run your test in both alphabetical and random order since running your tests in a randomized order is really helpful for finding hidden dependencies between your test methods.

Or you might want to run your tests using more than one sanitizer, such as both address sanitizer and thread sanitizer.

And I'll explain a little bit more about what these are later on if you're not familiar.

Or you could even vary arbitrary command line arguments or environment variables each time you test. And this can be helpful if the code that you're testing needs to modify or fake certain things when you're testing such as using a testing version of your web server or maybe mock data sets.

Xcode allows you to configure various options about how your app is run using the scheme editor today.

You can visit the Arguments, Options or Diagnostics tabs to control various things about how your app is launched.

But this only allows you to run your app interactively a single time using whichever settings you pick. What we really like and what we've been talking about is the ability to run our tests multiple times. And for that, we're introducing a new feature in Xcode 11 called test plans. So test plans allows running your test more than once with different settings.

Using a test plan, you can define all your testing variance in one place. And then, you can share that between multiple schemes.

Now, if you've previously duplicated your schemes just so that you can run your test more than once, you might be able to undo that and consolidate your schemes back down to just one using a test plan. So test plans is supported in Xcode as well as in xcodebuild for use on continuous integration servers and in Xcode Server.

And it's really easy to adopt in existing projects.

So rather than keep talking about it, I'd like to go back to my demo project and show you how it works. All right.

So back in the project that I've been working on, I'll get started showing your test plans by clicking on the New Test Plan File in my project.

So here, I can see all the details about my test plan.

I can see all of the tests organized first by test target then by test class and then by all of the test method inside of each class.

Now, using this view, I can see a few different things. I can see the full list of all of my tests, of course, or if I want to find a specific test, I can use it-- the filter field to search for it.

Or if I ever need to temporarily disable a test for any reason; for instance, if this one is not working at the moment, I can just uncheck it in enabled column. I can also modify settings related to test targets by clicking on the options button on the right.

And in this case, I happen to know that this test target is my UI test target and it would really benefit from running all of its test in parallel on multiple clone simulators so that they'll run a lot faster. So I'll do that by enabling that here. So next, I'll go to the Configurations tab of my test plan. And this is where I can control how my tests will run and how many times that they'll run.

On the left, we have a list of what are called test configurations. And we also have an item at the top called Shared Settings. And the Shared Settings is where I can control options for-- that are common to all the times in my test run.

And if we look at what all I can control in a test plan, there's a lot of things I can set.

I can modify different arguments for how my tester should be launched. I can modify settings about localization or alter how long my UI testing screenshots are preserved.

I can modify the test execution order, enable code coverage or enable things like run time sanitizers or memory diagnostics. Now, you may notice that some of the items here are in bold text such as the Environment Variables row. That signifies that I've given it a custom value. And here, I've given it a custom environment variable to run my test with.

I've also customized my tests to always run them in a random order instead of alphabetical. So what I really like to do in this test plan is to modify it to run similar to the example I gave earlier so that it will run two times using different languages.

I'll do that by adding a second configuration. And because I'm going run it in a different language, I'll give it a custom name of the language I'll use which is German. And while I'm here, I'll customize the name of the first configuration to US English. Now, in the US English configuration, I'm actually going to leave it as is with all of its default values plus the shared settings.

But in the German configuration, I will customize the language and the region. And I want to point out that since I'm editing the-- a configuration right now, in the pop-up menu, I see an extra item at the top labeled Plan Default Value. And this item represents the value that it would be inherited from the shared settings level of the plan. So if I ever want to revert this customization and go back to the inherited value, I can just select this. All right. So, now I've configured my test plan how I want it. Next, I like to show you a few ways that you can use test plans throughout the rest of Xcode. I'll go to an event-- a unit test file I've been working on called EventTests, which tests the event struct in my app.

And it's just got a couple of small unit test about the struct.

And if I click on any of the test diamonds in this test file, because I've configured my test plan with two configurations, it will run my tests two times total.

And that's great for situations when I want to run all of my tests especially on my continuous integration server. But as I'm developing my test, I may not run-- run them all those times. So I can choose to just run a single configuration by option clicking on the test diamond and here I see a menu with just so I could select one configuration.

I can-- Yeah, thank you . I can also do this in the test navigator. If I control click on any test in the test navigator, I see a similar menu allowing me to run either all the configurations or just one. And while I'm here, I also want to mention that the test navigator now shows the-- which test plan is actually active at the moment. I only have one test plan in my scheme but there can be multiple and we'll talk about the benefits of doing that a little bit later on. All right.

So now, I like to just run my unit tests in all of the configurations.

And now, we'll see Xcode quickly build my entire project and then run my test two times in the simulator. And it's already done.

So there, it looks like there was at least one issue though. So let's go see the details about that.

I'll see those details by clicking on the report navigator and going to the most recent test action.

So here, we can see that most of my tests succeeded. So they get the green check mark and one of the tests had an issue of some kind, so it has a red icon with a dash through it. And the test report is really good at showing me all of the details about everything that happened in my testing session. And if I expand this, I can see-- it looks like this test method succeeded in one configuration and failed in another.

And if I open it further, I can see exactly the issue. It looks like this test method received English text but it was expecting German text. So that looks like a real issue that I'll need to go back to my-- either my app code or my test and adjust and I'll do that later on.

But before I wrap up, I want to show you a few other enhancements to the test report here. If I want to only see test methods like this one which had a mixed status between the two configurations, I can do that easily by just clicking on the Mixed button in the scope bar. Or if I only want to see the results from a certain configuration, I can just click on the test configurations popover and select one of the configurations.

All right. So that's a quick tour of test plans in Xcode. And now, let me just go back to slides.

So, now that you've seen test plans in action, I'd like to mention a few details about how it works.

A test plan file is really just a JSON file with a .xctestplan file extension. And it contains all of your tests to run as well as all of the test configurations which describe how your test will run. Now, a test plan file is included in your regular project structure and it can be referenced by one or more schemes. Now, a test configuration meanwhile describes a single run of your entire plan's test. Now, each test configuration has a customizable unique name and it's a really good idea to give each test configuration a meaningful name for your project since we'll see that name in places like the test diamond popover menu and the test report like we saw. Now, each test configuration includes all of the options for how to build and run your test. And they can inherit any common options from the shared settings level of the plan.

So if you have any settings that are the same each time you run your tests, you can just define them in one place and not need to repeat yourself. So if you're curious, here is the full list of all the options that you can set on each test configuration.

And all this can be found in the test plan editors configuration's tab that I showed. So you might be wondering, how can I begin using test plans. If you have an existing project, you'll first need to convert it-- your scheme to use test plans.

To do that, first, edit the scheme then go to the schemes test action. And there, you'll see a button labeled convert to use test plans.

Clicking on this button will show a sheet offering the different ways that you can convert the scheme. Well, the first option is to create a brand new test plan file from the scheme's existing settings. If this is the first or the only scheme that you're going to convert, this is probably the choice you want. But another option is to choose an existing test plan in a project.

And if you choose this, it will show a sheet allowing you to pick an existing test plan in the work space.

And this is a good option if the scheme you're converting-- if you've already converted one scheme to use test plans and you want a different scheme to share that same plan.

And you can also use this if you've created a test plan file from scratch. OK.

So now that we've covered what a test plan is and you've seen how to begin using one, I'd like to offer a few potential ways that you could use test plans in your own project. So here is one basic example of a test plan that you could create.

Each of the red boxes represents one configuration in the plan. And the first one has the address sanitizer enabled and the other has thread sanitizer. Now, if you're not familiar with these, sanitizers are tools built into Xcode that instrument your code and help identify bugs that can be really hard reproduce manually.

And some sanitizers like these two can all be combined with each other. But if you construct a test plan this way, you could still run your test using both of them and get all of their benefits. Now, to get even more value out of this test plan, if your project includes C or Objective-C code, you can expand the plan by enabling the undefined behavior sanitizer in each of the configurations as well. So you might look at this and notice that the undefined behavior sanitizer is set in two places. It's repeated in each of the configurations. This would be a great setting to move up to the shared settings level of the plan instead.

And then, it will be automatically inherited by every configuration in the plan. Now, one thing to be aware of is that if you configure a test plan like this with mutually incompatible sanitizers is that if you run both configurations in the plan, Xcode will need to build your project twice, once for each set of sanitizers.

This is ideal for continuous integration environments where you don't mind your test taking a little bit longer to build since they're performing more thorough testing. But test plan isn't just about picking different sanitizers. As I showed in my demo earlier, you can also configure a plan with configurations representing different languages or locales.

For example, I've picked the US, South Korea, and Italy here and there is no limit on the number of configurations you can have. Now, if you configure a plan like this, you could use it to run your UI tests. And then, you could collect the screenshots from those tests by enabling the new localization screenshots feature in Shared Settings.

Localization screenshots is a new feature in Xcode 11, which will cause your UI tests to preserve all of their screenshots even for test that succeed. And it will gather data about the localized string that your app uses.

So this allows you to reference the screenshot for context when you're localizing your app or if you've already finished localizing, you could use the screenshots in your app store listings around the world.

Now for more information about this and other localization enhancements this year, check out our the great-- Creating Great Localized Experiences session on the WWDC website. Now, finally, I'd like to emphasize that you can mix and match these settings in a test plan however it makes sense for your project.

For example, you could construct a test plan like this one with three very different configurations.

The first one focuses on memory safety, and so it has the Address Sanitizer and the Zombie Objects memory setting. The second one is all about concurrency and it's got the Thread Sanitizer, the Undefined Behavior Sanitizer, and it runs tests in a random order each time. Then the last configuration is set up to collect extra diagnostics while running test. And it does this by setting a custom environment variable that the code being tested knows about in order to trigger more log collection.

And it also enables the option to keep all custom file attachments even for tests that succeed. So this is a fairly complex example but hopefully it shows the power and the flexibility of a test plan to run your tests however you would like.

So that's test plans, a new feature in Xcode 11 to get more value out of your tests by running them multiple times in different ways.

Now, I'll hand it over to Ethan to share some ways you can use Xcode for continuous integration.

Thanks, Stuart. To leverage the full power of test plans, you likely want to run your tests under many different configurations. A great place to do this is in continuous integration which automates the process of building and running your tests. While at your desk, you focus on getting individual tests to pass, continuous integration runs all of your tests across all of your devices giving you maximum coverage.

When it comes to continuous integration with Xcode, there are two primary solutions to choose from. The first is Xcode Server which is built directly into Xcode. With Xcode Server, you can easily set a box to build and test your app with minimal configuration. The second option is to build your own continuous integration setup. While more advanced, this option is ideal if you have custom requirements or need to integrate with existing infrastructure. If you do require a custom setup, you're in luck. Xcode comes with powerful tools that you can use to build your own automation. In this section of the talk, we're going to focus on option two and learn how to build a completely custom continuous integration pipeline. End to end, our pipeline will be composed of four steps involving the use of different tools at each step.

In step number one, we'll build our tests on a dedicated builder machine. In step two, we'll take the tests that were built and execute them on a suite of devices.

These devices will be connected to a second machine that we've set aside for running the tests. The first two tests will produce a set of build and test results.

These results will serve as the source of data for our next two steps. In step three, we'll mind the build and test results for failures in order to populate our favorite issue tracker.

And finally in step four, we'll track our code coverage over time to get a sense of how we're doing in terms of our overall test coverage. Let's start with the first two steps-- building and running our tests. For these tasks, we'll be using xcodebuild. Xcodebuild is the command line interface to Xcode that will power the core of a workflow. Behind xcodebuild, sits the xcodebuild system and the XCTest testing harness. With xcodebuild, there are two ways to run tests. The first is to build and test in the same invocation.

For this, you use the test action, passing in the name of the project and scheme you want to test, as well as the destination on which tests should be run. The second is to build and then test. This puts up the actions of building and testing into two separate invocations of xcodebuild.

One of the primary use cases of this functionality is to have one machine dedicated to building and another dedicated to running tests, which is the workflow that we're trying to achieve.

To accomplish this, you first invoke the build for testing action passing in the same parameters as before.

This will produce both the build products that are necessary for testing, as well as an xctestrun file.

The xctestrun file is a manifest that describes the build products and instructs xcodebuild what to do at test time. Next, invoke the test without building action, passing in the xctestrun file that was produced earlier.

This will cause your test to actually execute. It's actually possible to craft your own xctestrun files giving you more control over what happens during test without building.

If you like to learn more about the format of these files, check out the man page.

Also, note that the format can change between Xcode releases. In general, use the same version of Xcode, both for building and running your tests. Speaking of running tests, xcodebuild supports testing on multiple devices or simulators at the same time.

This can give you maximum coverage across a variety of device types and sizes.

And this is especially useful for UI tests since your app's UI likely varies according to size class.

If you'd like to learn more about xcodebuild support for multiple destinations, check out the What's New in Testing session from 2018. So far we've covered the basics of xcodebuild, how to build and run our tests. Now I'd like to talk about a couple options that are specific to test plans.

If you have a scheme with multiple test plans, you can list them all using the show test plans option. Having a scheme with multiple test plans opens up some compelling workflows. For example, you could have a long running test plan containing your full suite of tests, and a short running test plan with a handful of smoke tests.

If you do decide to have multiple, one of those test plans is considered the default which you can configure in the test action of the scheme editor. The default plan is the one that xcodebuild will run unless you tell it otherwise. To overwrite the default test plan, use the test plan option passing in the name of the particular plan that you want to run. Armed with all that we've learned about xcodebuild, we can start to fill in some of the gaps in our pipeline.

Starting with the builder machine, we'll use xcodebuild build-for-testing to produce the build products and xctestrun file that we need. This will get passed to the runner machine, which invokes xcodebuild test-without-building to execute the test on our suite of devices. The product of these two steps is the build and test results, which brings me to the next thing I'd like to talk about, and that is result bundles. We have some really exciting stuff to share with you about result bundles this year. First off, what is a result bundle? A result bundle is a file produced by Xcode containing structured data describing the outcome of building and running your tests.

It contains assets such as the build log, revealing which targets and source files were compiled.

The test report showing you which test passed and failed. The code coverage report reviewing which code was covered by the test ran, and any test attachments that were created by the tests using XC test attachment APIs. So how do you produce a result bundle? Just pass the result bundle path option to xcodebuild. Now that we know how to produce a result bundle, we can fill in another missing piece in our story. We'll add the result bundle path option to our Xcode build invocation to start producing those build and test results. Now, result bundles have been around for a while but in Xcode 11 we completely redesigned the underlying file format, which has come with several benefits.

First, the new format is highly optimized to be efficient on disc. In our own testing, we found result bundles to be four times smaller on average compared to the previous format.

This is especially useful in continuous integration where result bundles can be generated and stored at a very high rate. Second, it's now possible to open result bundles directly in Xcode allowing you to easily dig into the results of an integration. And third, for the first time we are providing a way to programmatically access the contents of the result bundle which we will be leveraging in our own continuous integration setup.

To open a result bundle in Xcode, simply double click the file to view its contents using the UI that you're already familiar with.

See failing and passing test in the test report, dig into the build failures in the build log, and see how you're doing on coverage in the code coverage report.

Thank you. To access the result bundle's contents programmatically, you can use a new command line tool in Xcode 11 called xcresulttool.

Xcresulttool gives you complete access to the structure data contained in the result bundle.

It emits this data as JSON and the format of the JSON is publicly documented and versioned. We'll be leveraging xcresulttool in our next step, which is to populate our issue tracker with any failures that occur during building or testing. To extract build failures, invoke xcresulttool using the get command passing in the path of the ResultBundle.

In the JSON output that follows, you can find build failures nested in one of the objects.

Each build failure includes both the failure message as well as the source file and line number that failed to build. Test failures are also nested inside of the JSON with the name of the test that failed, as well as the assertion message. Don't worry if you didn't follow those steps 100%. Like I mentioned previously, a big benefit of xcresulttool is that the JSON it produces is publicly documented. In fact, the tool itself can describe the schema of the JSON using the format description command.

The schema lists all of the possible types of objects that can be present in the output. So I encourage you to refer to it as you write your own automation on top of the tool. Last but not least, check out the tool's man page for more information. With our newfound knowledge of xcresulttool, we can now throw in step number three. We'll use xcresulttool get to extract those build and test failures from the result bundle and put them in our issue tracker. We've come a long way at this point but we have one last step to accomplish. We want to track our code coverage over time to know if it ever decreases. For this, we can use another command line tool called xccov.

Xccov provides programmatic access to the code coverage report either as human readable text or JSON.

To view the coverage report using xccov, invoke the view command passing it the result bundle. In the output that follows, you can see the line coverage for every target, source file, and function or method in your project. Now, simply viewing the coverage report may not be exactly what you want. If instead you want to compare two reports to see if coverage has gotten better or worse, you can use the diff command.

Pass the paths to two result bundles of the tool which will produce output similar to this.

In this example, we can see that the code coverage for AppDelegate file increased by 50% between the two result bundles. Like xcresulttool, xccov also has a man page, so check that out for more info. And with that we can finally fill in our last step. We'll use xccov to extract code coverage from the bundle to track our progress over time.

Awesome. Our continuous integration workflow is complete. Using xcodebuild to build and run our tests, xcresulttool to extract, build, and test failures, and xccov to view code coverage, we built a fully functional end-to-end pipeline that automates the testing of our app. Hopefully this gives you a sense of just how much power and flexibility is available to you using the tools that come with Xcode. We've only covered one possible workflow but with these building blocks the sky is really the limit. We've covered a lot of content in our talk today so let's briefly recap what we've learned.

We started off today's talk with an introduction to testing in Xcode. We learned how to write unit and UI test using XCTest and how to run them to catch bugs.

Next, we learned about Test Plans, a new feature allowing us to better organize our tests, as well as run them multiple times under different configurations.

And finally, we learned about the tools that we can use to build a custom continuous integration pipeline. If you'd like to learn more, grab a copy of these slides from developer.apple.com and be sure to check out the release notes for Xcode 11.

Finally, if you're interested in hearing about new APIs in XCTest for measuring the performance of your code, check out the Improving Battery Life and Performance session later today.

Come see us in the labs and have a great WWDC. [ Applause ]