Diagnose performance issues with the Xcode Organizer

Hello and welcome to WWDC.

Hi, everyone. My name is Shefali and I'm a performance tools engineer here at Apple. Today, I'm excited to talk to you about the Xcode Organizer.

Your apps are the backbone of our software experience. Now more than ever, it's important that we continue working together to improve battery life and performance so users can get the most out of their devices. Year after year, you will strive to build the most performant apps for your users, asking us what more you can do and what areas you should prioritize.

To help you ensure that your apps are performing well on customer devices, we're dedicated to providing you with actionable tools and data so you can quickly determine what areas of your app to focus on. The Xcode Organizer Window allows you to track aggregated battery, performance, and IO metrics for your applications. Today, I'm very excited to introduce some of the new features we've added this year. First, I'll go through a brief overview of the Xcode Organizer, and show you what it looks like in Xcode 11. Then we'll jump into the new features we've added this year, which includes a new category of metrics and diagnostics reports, as well as a new UI.

Finally, we'll tie all of these new features together in a demo where I'll show you how you can use this powerful tool to track the battery life and performance impact of your app from release to release, and we'll wrap it up with some next steps.

Let's begin with a brief overview of the Organizer.

What is the Xcode Organizer, and how do I get to it to see my app's data? As the name suggests, the Xcode Organizer Window is built directly into Xcode, which means all you have to do is navigate to the menu bar, go to "Window," and click "Organizer" to launch. What's great about using the Organizer is that it requires absolutely no changes to be made to your app. As users use your app, we collect battery life and performance data from consented devices. This data is then aggregated on our servers and sent back to you through the Organizer.

To learn more about the Xcode Organizer, I recommend you check out the "What's New in Energy Debugging" talk from 2018, and the "Improving Battery Life and Performance" talk from 2019. Last year, through the Organizer, we provided data for various app-usage metrics. Since then, we've loved seeing you use the metrics and diagnostics reports to monitor your app's performance and drive some really great bug fixes.

Now let's take a look at the new data we've added in the Organizer this year.

This year, we're providing more data across both metrics and diagnostics.

We've added a new category of metrics for scrolling. We're calling these "hitches." Before we go into the details, let's take a look at the Meal Planner app I've been working on with some colleagues, and what a user perceives as a hitch.

Notice the slow, skipping, and jittery scroll. And here's the same application, but without any hitches.

As we can see, there's quite a difference between the two experiences, and this smooth scroll here is what we wanna help you achieve using the scroll hitch metric.

But what is a scroll hitch? A scroll hitch is when a rendered frame does not end up on-screen at its expected time while a user is scrolling within your app. This usually causes frames to be dropped, which can be perceived as a jittery scroll.

For the scrolling section of the Organizer, we take the hitch time, which is the total amount of extra time frames take to show up on-screen, and we divide that by the scroll duration, which is the total amount of time a user spends scrolling within your app.

The result is the hitch rate, which is an important metric you can use to measure app performance because it provides an estimate of the severity of hitches as perceived by the user.

There are three buckets for how you should interpret hitch rate.

This first is when the hitch rate is less than five milliseconds per second. Here, there's quite a large window of time between dropped frames, and users are likely to perceive the smooth scroll that we saw in the Meal Planner app just a little earlier.

Between five and ten milliseconds per second, frames are dropped every couple of seconds. And when the hitch rate is greater than 10 milliseconds per second, users will be perceiving frame drops very frequently, leading to a poor scroll experience.

To learn more about hitches, I recommend you check out the "Eliminate Animation Hitches with XCTests" talk we've prepared for you this year.

The second piece of new data we've added this year is disk write diagnostics reports.

Last year, we gave you metrics for this category, and this year, we're providing you with even more context to help you pinpoint and really narrow in on any issues your app encounters.

Disk write diagnostics logs are aggregated when disk writes exceed one gigabyte in a 24-hour period.

By reducing the amount of writes your app is doing, you can ensure better performance, longer battery life, and overall good device health. Let's take a look at an example of what these reports look like.

In the left pane of the Organizer, when we navigate to the "Reports" section and click on "Disk Writes", here's what we'll see for our app. We'll go into more detail about this during the demo.

So far, we've seen some great new data, and to go along with it, in Xcode 12, we've completely redone the interface. We're thrilled for you to finally see it. The new interactive UI allows you to compare and contrast metrics for different versions of your app with a single click. Let's go to the Organizer and take a better look at all of these new features.

Let's say I'm a developer for an app called Meal Planner which allows users to plan meals for the week by saving pictures and recipes so that they're all in one place.

In XCode 12, when I open up the Organizer, I can do a side-by-side comparison of two of my releases thanks to the new interactive UI.

I can now click on any of the older versions of my app...

and on the right side, we can see the UI update to reflect the differences between the latest and selected versions. There's also a more detailed breakdown of the subcategories accounting for the battery usage of my app. We can see here that in the latest version, there's an increase in Camera and Bluetooth usage that I now know to go investigate. I can also go back and select any of the older versions whenever I want.

This is really useful because it opens up a lot of new possibilities for version comparison, and it gives me control over the data I'm looking at. As developers, we love seeing data about the usage of our app, but what if it takes days or even weeks before we can see it in the Organizer? It's important to note that last year, we set a threshold where we expected each app version to have a certain amount of usage from user devices before we could show histograms for them in the Xcode Organizer. This year, we're thrilled to show this data even sooner, giving you an early glimpse of your app's performance. In Xcode 12, we've lowered the required usage threshold by a factor of five. This means that if you're currently able to see data for your app, you'll now see this data even sooner. If your app's usage was below the old threshold but is above the new one, you'll start seeing data for the first time. We're incredibly happy that a much larger audience of app developers can now use the Organizer to monitor their apps' battery life and performance impact. Looking at this battery-usage graph, we can see an icon in the X-axis by version 1.0.1.

This icon indicates that a version of our app had limited usage. Associated with this icon is additional information regarding the margin of error for that version. We can see this on the right side here.

This is important because we're analyzing fewer data points for these versions with limited usage.

The data from the smaller working set will stabilize as more people update their app and use the new version. As the usage increases, the error margin will decrease until the version gets enough usage where the error margin is low enough that's it's omitted from the UI.

Now let's take a look at how we can use the new reports as part of our workflow.

Clicking on the disk write metrics on the left, we can see that there was a sudden increase in disk writes in version 2.2. Let's walk through how we at Meal Planner would triage this regression. After identifying the spike in disk writes here, we'll navigate to the "Reports" section on the left...

and click on "Disk Writes".

It looks like there were a couple of different issues contributing to the increase we saw. The top one looks like it's actually related to a new feature we added in version 2.2, where users can now add their own custom pictures and recipes to their meals. Since the stack trace tells us exactly what the signature of the issue is, we can jump to that part of the code and begin investigating.

Under the signature, we can also see what percentage of our disk writes a given signature accounted for.

For a particular signature, the "Reports" section on the right shows us a breakdown by device type and operating system, and the bar graph below the pie chart gives us a trend of how many reports were received for our app over the last 14 days. Similar to the energy reports, we also provide sample stack traces per signature and additional information regarding the stack trace in the "Details" section on the right.

I can use all of this information to make the fix so that our app isn't doing as much writing. I'll go ahead and mark the signature as resolved so my colleagues can address some of the other regressions the Organizer caught. Once our users are using the version with this fix, we'll come back to the metrics to confirm that our data returns back to what it previously was.

We're incredibly excited to show you more data in a new interactive Organizer, and we're excited to see how you use all of these new features to drive some great optimizations within your app. If you're a developer using the Organizer, I encourage you to go compare metrics for two of your app versions with the new interactive UI, check out what scrolling performance looks like for your app with the hitches metric, and take a deep dive into the disk writes for your app by using the disk write reports along with the disk write metrics. We're very excited for you to continue using the Xcode Organizer to optimize your apps for battery life and performance.

Thank you for taking the time to listen to this presentation, and I hope you enjoy the rest of this conference.