Take an advanced class in Testing and Xcode Server. You'll gain an in-depth knowledge on the lifecycle of a test, how they're are hosted, and how using modern observation can help you make bulletproof tests for your app. Then, learn about changes in Xcode Server that make continuous integration easier than ever, including configuring your own user for testing, enhancements to issue tracking, email notifications and support for crash logs.
My name is Zoltan.
Later my colleague, Eric, will join me on stage.
We're both engineers with the Xcode team.
Software development these days is a lot
like conducting an orchestra.
You and I maintain suites of tests, and when one
of them is not performing well,
we need to quickly understand the issue and correct it.
Okay. Maybe you're a testing maestro with hundreds
of thousands of tests.
Well, Xcode has tools and techniques
to conduct even the largest test suite,
and today we're going to show you how.
First, we're going to introduce some concepts
that haven't been addressed in previous sessions, and then,
we'll introduce some new features in Xcode,
in Xcode Server, in particular,
the configurable integration user.
And then, we'll conclude with some new features in xcodebuild.
Let's get started.
So, as a brief recap of testing, you can think of testing
as these four characters.
There's XC test.
That's the framework for your tests
for both Objective-C and for Swift.
And your tests are compiled to bundles before they're run.
Xcode, that's the IDE for authoring your tests,
and you can also run individual test while you're developing.
Xcode is also where you review reports
from both local test runs and from Xcode Server.
And talking about Xcode Server,
that's the continuous integration solution
for your tests.
You set up bots to periodically run your tests,
and Xcode Server will generate reports for you.
And if something goes wrong, it'll notify you.
There's a great way to keep track of your project over time.
So, Xcode Server is built on top of xcodebuild,
and you can use xcodebuild on the command line, too.
You can run tests and see results in the console.
Xcodebuild is the building block for custom,
continuous integration systems, as we'll see later.
So, you can learn more about these four characters
in our previous sessions.
But, today, we want to look at some concepts
that we haven't covered before.
We want to take you behind the scenes
and show you exactly how your tests are running.
And to do that, it helps to think of a timeline.
After your tests are compiled, they must be hosted.
That gets the test started in the first place.
And then, once the tests are running,
you can see detailed progress
in your tests using a technique known as observation.
So, we're going to look at these two concepts,
hosting and observation.
Let's look at hosting first.
So, the hosting story, it's different depending
on whether you have unit tests or UI tests.
So, for unit tests, your test bundle is loaded directly
into your application.
And, in this case, we call your application the
But for your UI tests, your test bundle is going to get loaded
into a UI test runner which is separate from your application.
And in this case, we refer to your application
as the target application.
Now, this has some implications.
For your unit tests, you will have direct access
to your applications data structures and API,
but for your UI tests, you have
to access your application using accessibility and send events
and see your application as a user would from the outside.
For your unit tests, all your tests are going to run
in the same launch of the host application.
So, you should be careful to clean
up between test invocations.
But for your UI tests, your tests can terminate
and relaunch the application.
So, that's great if you want
to test how your application starts up.
Okay. That's hosting.
That's how your test gets started in the first place,
and once your tests are running,
then you can use a technique known as observation
to see detailed progress in your tests.
So, let's zoom in on that timeline.
Here you can see two test cases, one running after the other.
And these test cases belong to a test suite.
Test suite corresponds to a test class you've written.
And you'll have multiple test suites in your test bundle.
And maybe you're interested
in doing some setup work before any of these tests run.
Or perhaps doing some tear down work
after all the tests have finished.
Or maybe you're interested
in doing some custom logging while the tests run.
Well, you can do that with the XCTestObservation Protocol.
You write an object that conforms to this protocol,
and after registering it with a shared observation center,
your object will receive call backs.
So, for instance, before any of the tests start,
you'll get this BundleWillStart call back.
And then, before the suite starts,
you'll get a SuiteWillStart call back.
Then, for every test case that runs,
you'll get this testCaseWillStart call back
and testCaseDidFinish call back.
If something goes awry,
you'll get this testCaseDidFail call back.
And then, as the tests are wrapping up,
you'll get this testSuiteDidFinish call back,
and your final chance to do any work is
in this testBundleWillFinish call back.
So, here's an example.
This object conforms to XCTestObservation Protocol.
In the object initializer, I'm going to register
with a shared observation center.
Then, as the tests progress,
I'm going to log events, such as this one.
When things go wrong with the test, I'm going to log this.
And then, once the show is over, I'll log that, too.
So, an ideal place to set up this observer is
in your test bundles info.plist.
So, you do that using Xcode's info.plist editor.
You add this NS principal class entry,
and this is a test specific instantiation for your observer.
It's independent of the class load initializer.
Okay. That's hosting and observation.
They're useful concepts to help understand exactly how your
tests are running behind the scenes
and to help you diagnose issues with your tests.
Let's look at some new features in Xcode 8.
Crashes are a frequent source of failures in tests,
and the crashes can be in both your host application
and the target application.
So, normally, Xcode will relaunch your host application
to complete your test suite.
But it's up to you to gather the diagnostics necessary
to resolve the crash.
I'm please to say that this year, Xcode is going
to help with this issue.
We will now gather the crash logs for you in the test report.
So, this is both for UI and unit tests, for local
and server runs of your tests.
The crash logs will be collected in the test report.
And you can see the textual representation
of the crash there, or you can choose to see the crash
in the context of your source.
And I'd like to show you that now.
Eric and I have been moonlighting.
We've been working on a tvOS application in our spare time.
So, I'll run the application
by control clicking on the Run button.
It's an application defined nearby concerts
with some strangely test related band names.
I can move in the simulator using the keys
on the keyboard here.
So, we're adding a new feature to this application
to support users who have location services disabled.
Let me show you what it's like for those users.
I'll go to the settings menu, and in privacy,
I'll disable location services.
So, now, back in Xcode, we've added a new ViewController
to support those users.
They can enter in a zip code.
Let me just add one more test for this ViewController.
So, I want these tests to run in a scheme on their own.
So, I'm going to go to the scheme menu,
and I'll duplicate this existing scheme.
I'll call this something meaningful.
And, in the settings for the test action,
I'm going to disable the debugger.
So, that's a technique to avoid interrupting the tests.
It allows all the tests to run to completion
without breaking into the debugger.
And it's similar to how your tests run on Xcode Server.
So, I'll disable tests that are not related
to location services.
I'll share the scheme.
And then, I'll run the test with Command U.
So, here's a new ViewController for users
who have location services disabled.
They can choose to enter a zip code.
So, right now, the tests are entering a zip code,
and that's an unexpected crash.
So, right now, Xcode has gathered the crash log,
and it's reported the test failure.
And, let's have a look at that.
So, here's the failed test, and I can click
to jump to the test report.
I'll disclose the test transcript.
Here's the events as the tests were entering in the zip code,
and here at the bottom is the crash.
Now, I can click to see the textual representation
of this crash, but in this case, I want to show you the crash
in the context of the source.
So, I'll click on this arrow here.
So, here's the exact line that the crash occurred on.
And you can see in the top left, the stack frames
in the debug navigator.
Now, I can see the crash in the context of my source
and diagnose the issue.
So, I happen to know that I'm referring
to the wrong ViewController here.
Instead of parent, this should be presenting ViewController.
So, I'll make that change, and now, I'll rerun the test
by going to the test navigator and clicking on this icon here.
So, earlier, I showed you the test report
for a local test run, but that report would look exactly the
same if it came from Xcode Server.
So, now the tests are reentering a zip code.
Great. So, these tests have passed.
We would now check in this tests and build confidence
that we're supporting users who have location services disabled.
But, for now, let's go back to slides.
So, you've seen how Xcode will now gather crash logs for you
and capture them in the test report.
And you can choose to see the textual representation
of those crashes, or you can see the crashes
in the context of your source.
So, that's a great way to diagnose the issue
and to make the fix right in the context of your source.
So, we have some new features in Xcode Server,
and to show you more, please welcome my colleague,
Eric Dudiak, on stage.
I'm Eric Dudiak, and I'm going to talk to you a bit
about Xcode Server and what we have new in Xcode 8.
So, let's go over a little bit of an overview
of what we're going to talk about today
in Xcode 8 with Xcode Server.
So, we have custom environment variables
that you can now set per integration.
We have advanced trigger editing workflow
that we've improved in Xcode 8.
We also have some enhancements to issue tracking and blame
to make sure you get notified of issues
as they come up on your bot.
And we'll see how that plays with upgrade integrations,
a new feature we have for you.
And finally, we'll talk
about the configurable integration user
which is new in Xcode 8, as well.
So, let's jump right in, and let's talk
about custom environment variables.
Now, this is a little bit of a cheat.
It's actually new in Xcode 7.3, and it allows you
to configure the exact environment that's passed
to xcodebuild on your bot.
And it controls how your integrations are run there,
so you can configure any number of settings that you might need
for your bot to run on your server different
than you might have locally.
This is a great way to customize how your server runs your
integrations, and it's a great way to do that without having
to create a lot of extra schemes in your project.
Now, on to what's actually new in Xcode 8.
So, we've significantly improved the trigger editing experience.
So, we have two types of triggers, scripts and emails.
First, let's talk a little bit about trigger scripts.
These run either before your integration
or after your integration, and they're normal shell scripts.
So, we've improved the editor by giving you a lot more space
to see the scripts, and you can actually see exactly what's
in them, so you have a lot more visibility into that.
Along those lines, we also now let you name triggers.
This is great if you work on a team.
Your other teammates can see exactly what each trigger is
supposed to be doing, and if you have particularly long running
scripts that are part of your integration triggers,
you will see that called out in the status
as Xcode's integrating.
So, when you see a project integrating into status UI,
you can actually see which script it's currently running.
Finally, if you have a script that you come up with later
that you actually really wish was running before all your
other scripts, you can now reorder triggers in this UI.
Simply add a new script trigger, drag it to the top,
and it will become the first to run on your bot.
Now, let's talk a little bit
about the other type of trigger we have.
We have email notifications.
Historically, in Xcode Server,
this was always a one email per integration setup
which can leave you with a lot of spam.
So, in order to help you and reduce the amount of email
in your inbox, we've split these types of triggers
up into two different types.
We still have the report triggers or report emails
that come out every time you run an integration.
Of alternatively, you can now schedule them
to run just once every day or once every week.
This way, no matter how often your bot runs,
your email isn't flooded,
or your inbox isn't flooded with a lot of emails.
And we think this is great for managers who want to check
up on the health of a bot continuously
but don't want all their inbox to be filled with emails.
Additionally, we also, now,
let you configure certain fields on the email.
So, you can configure your cc field as well
as your reply to fields.
This lets you have a lot more control over exactly the types
of emails you're sending out.
Now, I hinted a little bit earlier that we actually, now,
have two different types of email triggers.
The other is issues.
So, new issues come up.
We will now send you an email to the people of interest
for that particular issue.
We'll go over a little bit of that in a minute.
But, if you do have more committers that are part
of the code that you're integrating
than you necessarily want to email,
we do let you filter recipients right here
to make sure you're emailing just the people you want to be.
So, if we take a look at that.
If you have multiple repositories, you can choose
to only send emails to committers
from certain repositories, or if you know exactly
which domains all the email addresses will be
from that you want to send emails to,
you can add those right here.
So, let's take a minute and talk a little bit about issues
or build issues anyways.
And, nobody is perfect and writes perfect code every time.
That's the whole reason we have continuous integration.
That's why we have unit tests.
We have unit tests because we know
that they will inevitably fail.
That's why we write them in the first place.
We also, occasionally, commit code that just doesn't build,
and some of us don't check it before we commit,
and that's exactly the kind of thing
that continuous integration is great at catching.
And, in these types of cases, Xcode will send you an email
and notify you that you broke your build.
But those aren't the only issues that can come up.
Sometimes, even if you write absolutely perfect code,
things can change around you.
That can be when you install a new Xcode, you get a whole bunch
of other new features, too.
You get new SDKs that might have new deprecations.
You might get new issues from language improvements.
And, we like to make sure that every Xcode
that we ship is smarter than the Xcode we shipped before.
So, you might see issues that we didn't find before
that have always been there that Xcode now tracks
such as static analysis issues.
So, the important thing when you see one
of these emails is we really want
to make sure it's actionable, and let you know
that you're receiving this email, for the issue emails,
when it is because something happened
that you can do something about.
So, the first type is if you introduced an issue.
And that's because you broke it.
So, this will call you out, as you see right in the email.
You'll see an email like this
that says you introduced an issue.
And we know this because the issue showed up on
or near a line that you just recently modified.
We also might know if, between the two integrations
where the issue showed up, you were the only person committing.
In that case, that's pretty much a fair guess
that it's you that broke it.
Now, that's not always the case.
Take, for example, a application that's built
on top of a framework.
A change in the framework might cause breakage
in the application without anyone committing
to the application.
So, in this case, you'll see an email a little bit more
like this, and it'll be a little bit less condemning of your work
and will simply say that you might be able
to help fix an issue on the bot.
And we know this because you commit frequently to the area
where the issue showed up.
So, we, essentially, assign ownership to various areas.
And, keep in mind, that this is a bit fuzzier matching
than when we're directly blaming someone.
So, expect this email to go out to a bit of a wider audience.
It casts a very wide net when trying to figure
out who might be of interest for a particular issue.
Now, that's great for issues that come up in your code,
but sometimes, you can actually get issues that come
up regardless of code changes.
One of the easiest ways to see that is actually
by reconfiguring your bot.
So, Xcode 8 or Xcode Server in Xcode 8 will now track changes
to your bot configuration and actually call those out.
Whenever possible, we will attribute any new issues
that come up to changes in the configuration,
such as if you enable testing or enable static analysis.
Those types of issues may have been in your code forever,
but we can actually attribute them specifically to changes
in the configuration of the bot rather
than a change in your code.
We also make sure to include this information in emails
for the next integration, so that when you do see one
of these emails, you know that some of these changes might be
because you're picking up a configuration change.
Let's go back to something we were talking about earlier
of installing a new Xcode and getting a bunch
of new features in the process.
Well, when that happens, Xcode Server
on Xcode 8 will actually reintegrate your entire project,
and we call this an upgrade integration.
We take the exact same revision of the previous integration
of all your repositories and simply rerun all your tests,
rebuild everything, rerun the static analyzer,
and when this happens, we know that the issues that come
up in your project at this point, any new issues,
are specifically because of the upgrade
since we took the exact same commits as before.
This saves you a ton of time trying to track down changes
in your source code that just aren't there
because the issues came up from changes around your code.
So, that's some of the new features and issues and blame,
but I want to talk a little bit about my favorite new feature
in Xcode Server and Xcode 8 which is our new,
configurable integration user.
So, we now give you full control
of the macOS user that's running your integrations.
And this gives you a lot of improved visibility
into how your integrations are being run,
and more to the point, it allows you
to configure exactly how your integrations run.
Historically, in Xcode Server, there's a hidden macOS user
that was running all of your integrations in the background.
You wouldn't have access to the password,
and you wouldn't be able to log in as them.
This meant that you were basically getting a stock user
no matter what, and you couldn't make any changes.
Now, you'll own and manage this user yourself.
You will be given the password.
This user will be a completely normal macOS user.
It'll be any user on the system.
Can be anyone you want.
We suggest a new user.
But it will available at login window,
and it's fast user switching.
So, you can log in to that user.
You will notice you're running as that user
through a menu extra that shows the Xcode Server
We'll see that in just a minute.
Let's go over how you would set up this user.
So, here I've opened the server app to the Xcode service pane,
and in order to enable the Xcode Server,
I need to go ahead and choose an Xcode.
So, I'll choose a new Xcode 8 that I have installed,
and I'll be presented with this dialog, asking me to set
up which integration user I want to use, and I'm going
to go ahead and create a new user just
for running integrations.
I'll go ahead and give it a name
and a password, normal user stuff.
And I'm going to leave it as a basic user.
And when I push "create user," Xcode's going to go ahead
and do a little bit of work
in the background while it gets everything ready
to run integrations, and once that user's mostly set up,
we're going to be asked to login as them.
Now, we are asked to log in as them because we're going
to go ahead and run a little bit of the setup assistant.
Remember, it is a real, normal macOS user.
So I can log in, I can sign
into a test iCloud account, for example.
I can stage any data that I want to have for my integrations.
Anything like that.
When Xcode's ready to run integrations,
you will see a notification like the one you see
in the top right here, and in the top right in the menu bar,
you'll notice that there is a little hammer indicating
that this is the Xcode Server integration user.
With that all done, I can go ahead and switch back,
and I'll be back in server app, and I'll have the indication
that the user is ready to go, logged in,
and integrations can begin running.
So, now, we've seen how to set up that user.
Let's take a look at some of the things that we can do now
that we have access to the integration user.
And for that, let's go to the demo machine.
Here we have that same project we saw earlier,
and we saw an issue where it was crashing
because it didn't have access to location.
And we had to go in manually to the simulator
and turn off location access to test it locally.
But, as we're developing our application,
we might not want to do that.
We probably want to leave location services turned
on on our Apple TVs and on our simulators,
so that we're running, more or less, the way most
of our users will see it and how we want to see the app.
But, now that we've fixed the issue we saw earlier,
we definitely don't want it to come up again,
and we want to be notified if it does.
So, it just so happens that this machine I'm actually
on right now, happens to be my server machine.
So, I don't have to go anywhere to go to my server.
I can actually just go to the fast user switching menu,
and I see that have a configurable, continue,
integration user called Xcode Server.
And I'm just going to go ahead and select that
and enter my super-secret password that's only four
And we will log in as this user.
So, I've gone ahead and changed the desktop background
so that I know that it's the build service user.
And, here we see the hammer icon of Xcode that indicates
that this is Xcode Server.
We see it's on and waiting for integrations.
So, it's all configured,
but I want to go ahead and go into Xcode.
So, I can actually configure any settings I want in Xcode locally
on this user, and they will be picked up in my integrations.
So, I'm going to use a neat little trick.
If I go to the devices menu, I see all the devices
and simulators that are plugged into this particular machine.
Now, being generous at Apple, we give you one
of every device we've ever made, but if that's not enough
for you, for absolutely free, you can get more simulators just
by clicking this Plus [+] button.
Now, the Apple TV 1080p simulator's the built in one
that I got when I installed Xcode.
I've gone ahead and created another one called Apple TV no
location, and I've actually got it booted here.
And if I go in, I can double check, and it is, indeed,
has location services turned off.
So, I go to settings, general, privacy.
Location services is turned off.
So, this simulator is all configured for use,
and my other one's still usable
for when I want location services turned on.
Now, let's do a quick switch back,
and just log back into my normal user.
Normally, I'd be walking across the room
to my actual work machine, so this is a little faster.
And here we see all the UI tests.
Now, I already have a bot that's integrating this normally,
but I want to go ahead and create a bot
that will integrate just the case
where we don't have location data.
So, I'm going to go product, create bot,
and use the Harmony no location scheme on this server.
Going to go ahead and give it location, sorry,
access to my repository so it can check out the project.
I'm going to disable the archive action
because this will be the same as my normal bot.
So, I don't care about that, but I do want to run tests.
And, let's leave static analysis on.
And I want to be notified as soon as I break this,
but probably once a day is good enough.
So, I'll know within 24 hours if we ever break this again,
our UI test will catch it on our server.
So, we'll just run it every day at 1:00 AM, and here I get
to select the devices.
By default, every integration runs on all devices of that OS.
So, this is a tvOS project, so it'll run
on all tvOS devices and simulators.
Instead, let's do specific TV devices, and I'm just going
to select the Apple TV no location.
This is that environment variable view we talked
I don't have any environment variables I need.
I'm going to go ahead and make sure that I get emailed
when a new issue comes up in this particular case,
and go ahead and create the bot.
So, that is going to go ahead and kick off an integration
which we see running here.
It's checking out building.
We've already seen the UI test run a little bit,
so let's actually look at an integration
that I prebaked a little bit earlier.
And if we go to the test,
we can see that it did run all our tests including this zip
And we can see, if I look at some of the screenshots from it,
it is, indeed, running without location because it's opening
that ViewController that we saw earlier.
So, we saw how you were able
to get significantly improved visibility
into how your user runs.
You can log in as them and see everything that's happening.
We saw how you could customize different settings
such as simulators to run exactly the operations
that you want to run on your integrations.
We saw this was a completely normal macOS user
that I was able to switch to in fast user switching
and that I had the password to.
And, we got to see that menu extra
that shows you a little bit of your integration status
in that integration user.
Now, with this great new power comes a little bit
of responsibility, and so we have some best practices
First, we highly recommend you dedicate a new user.
You want this to be as similar to your customer's experience
as possible, so you don't want all your settings
to be causing impact on your bots.
We also recommend you avoid administrator accounts.
Keep in mind that anyone who can create or edit bots
on your serve will have access to this account
through triggers, so this goes the same
for any private or customer data.
Try to avoid storing that in this user.
If you want to stay logged in as this user in the background,
use fast user switching.
Integrations can continue to run
in the background just as they always have.
However, if you do want to run the integrations
as the front most user,
make sure that you turn off screen lock.
Just like iOS screen lock blocks testing,
so does macOS screen lock.
And finally, make sure to customize this
for whatever needs you have.
That includes, as we saw earlier, any simulators.
If you have any specific networking configuration
that you need for your integrations,
any stage user data or settings that you want for UI tests,
go ahead and configure those.
And finally, if you have any advanced provisioning,
such as if your administrator gives you a provisioning
profile, however you would configure on Xcode,
you can actually configure
that in your configurable integrations user, now,
and make sure that code signing will work for you.
So, with that, I'm going to bring Zoltan back to talk
about some new features in xcodebuild.
Thank you, Eric.
So, you've seen new features in Xcode and Xcode Server,
and now we have new features to show you in xcodebuild.
xcodebuild has the test action
for custom continuous integration systems.
You give it a workspace, a scheme, and a destination,
and xcodebuild will dutifully build your sources.
It will install built products onto devices as needed.
It will run your tests, and then report results
to you on the command line.
So, this year, we're introducing two new options for this action.
With the only testing option,
you can effectively constrain the set of tests
that the action will run.
So, you can specify TestCases or TestSuites or TestBundles to run
to the exclusion of other tests.
And you can also use the skip-testing option
to specify TestCases to exclude
from the tests while running everything else.
And the big news this year is
that we're introducing two new actions to xcodebuild.
So, we're effectively splitting the test action in two.
These actions are already available in the Xcode IDE,
but we're going to bring them to xcodebuild, now.
And so, we'll look at each in turn.
Build for testing is just the building portions
of the test action.
So, you give it a workspace, a scheme, and a destination
as before, and it will build your sources,
perhaps making certain symbols visible,
and then it will output the built products to derive data.
It also produces this xctestrun file.
It's a kind of manifest
for everything your test needs to run.
We'll come back to that in a minute.
Test without building then, is the second part to the story.
You give it a workspace, a scheme,
and a destination, as before.
And, xcodebuild will find the build products in derived data.
It will install those onto devices as needed,
and then run your tests and report results just as before.
But the cool thing is, you no longer need
to provide a workspace.
Instead, you can just provide this xctestrun file,
and Xcode will run your tests from binaries alone.
So, it will ingest the xctestrun file.
It will find binary products relative to that file,
and then it will run the tests and report results
in exactly the same way as before
but without having any access to your sources.
So, this is ideal
for distributed testing environments.
You would build your tests on machines up to optimized
for building, and then move those built products
onto machines optimized for testing.
And then, in parallel, you can gather the test reports.
So, this is all made possible by the xctestrun file,
that manifest for your tests that specifies which tests
to run and which to skip and which test machines.
It provides environment variables
and command line arguments to your tests,
and you can read more about the format
in our man pages using this command.
So, we covered a lot of ground today.
We first reviewed testing in terms of these four characters,
and then we looked at some new concepts in Xcode.
They were great for diagnosing issues with your tests,
and then we introduced new features in Xcode,
in Xcode Server, and in xcodebuild.
I hope these insights and these new features help you
to develop test suites which perform well, and, after all,
that will help you deliver great applications.
You can read more about this session at this URL.
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.