Animations can dramatically enhance the user experience of your app, provide a sense of direct manipulation, and help people to better understand the results of their actions. Animation hitches can break that experience. Discover how to use XCTest to detect interruptions to smooth scrolling and animations, and learn how to catch regressions before they affect the people relying on your app.
Hello and welcome to WWDC. Hi everyone, my name is Tanuja Mohan and I'm a software engineer on the power and performance team at Apple.
Animations are an important part in our applications customer experience.
Animations can be subtle such as when we tap on the back button to navigate from one screen to another or they can be the main focus of a gesture, such as when we scroll up or down on an application. We want these gestures to produce smoother responses since it's noticeable when a navigation takes too long, or a scroll appears to jitter. We called these user perceptible jitters, hitches. A hitch is anytime that a frame appears on the screen later than expected. This is distracting to your users, and is detrimental to the perceived quality of your application. Let's zoom in on the individual frames of our application to see what is really going on. The first three frames are displayed as expected - we have a gradual movement of the list, perfectly matching the movement of our finger. But then frame 3 remains on the screen. The application no longer appears to track the movement of your finger. Then, when frame 4 is displayed to the screen, the list makes a sudden large jump back to your finger. This is not what we expect and what we want to avoid. To understand what is happening here, we first need to understand how frames are displayed to the screen. Frames on iPhones and iPads are usually expected to update at 60Hz, giving a cadence of 16.67ms for each frame. On iPad Pro, we can expect 120Hz updates, or a cadence of 8.33ms.
This cadence is represented by VSYNCs, which are when the screen determines whether to swap a frame onto the display or not. We see a hitch when a frame misses its expected VSYNC. The severity of a hitch is measured by how late the frame is to appear on screen. In this example, frame 4 is 16.67ms late. There are two ways we can quantify hitches. Hitch time is a time in milliseconds that a frame is late to display on screen. We prefer to express hitch ratio, milliseconds per second, which is the total hitch time in milliseconds over some other duration of time and seconds, for example over the duration of a test.
This sounds complicated, why don't we just say dropped frames, and measure frames per second? Frames per second is an absolute target that is easily skewed.
If your test contains any resting time during the execution of an animation then fps is useless since we did not expect any frames to be swapped during the resting period. And we often intentionally do not target the maximum fps. Maybe a game only wants to run at 30fps, or a video at 24fps.
For power and performance reasons the hands on the clock app icon only run at 10fps. Hitch time's target is always zero, and is reliable even with these considerations. But hitch time is not always comparable. Total hitch time for a 1 second test cannot be compared to that over a 10 second test.
By normalizing hitch ratio as milliseconds of hitches per second of test duration we get a metric that is both comparable across different tests and can be used as an approximation for the end user impact. For end user impact these are the target hitch ratios we recommend, and use in our tools.
A hit ratio of less than 5 milliseconds per second is good user experience.
In the 5 to 10 millisecond per second range users will start to notice hitches and these hitches should be investigated. At 10 milliseconds per second or more, hitches are quite distracting to the user and we should take immediate action to resolve them. In iOS 14, you can track hitches in both your development and production workflows using our suite of tools.
The XCTest framework allows you to collect hitch and animation data directly in unit and UI tests, while MetricKit and the Xcode Organizer give you access to performance metrics from your customers. In this portion of the talk we're going to focus on the development workflow of catching hitches using performance XCTests. If you want to learn about how you can view hitches in your production workflow, check out the separate talks we have for MetricKit and Xcode Organizer this year at WWDC 2020, as well. In Xcode 11 we introduced XCTMetrics. These metrics specify what part of the system you want to measure in your test. The XCTMetrics that we have available today allow you to test around clock time, CPU utilization, Memory-use, os_signposts, storage and with Xcode 12 we have a separate metric to measure application launch times. We also have a template for you to write your own custom metrics. In this talk, we will focus on XCTOSSignpostMetric which is the XCTMetric used to do animation performance testing. Starting with Xcode 11, you can use the XCTOSSignpostMetric to measure the duration of the os_signpost interval.
Now, with Xcode 12, when using an animation os_signpost interval, you will receive not just duration, but also 3 hitch-related metrics, frame rate and frame count. You may already be familiar with frame rate and frame count.
These two values measure the frequency and number of frames displayed to the screen respectively. And now you are also familiar with hitches. You can now track how many hitches occur in the tested code block, when the total duration is you spent hitching in your test, and the ratio of this total time hitching over the duration of the measured code block. To collect these metrics, you first need to instrument your code to emit an os_signpost interval.
There are three ways you can do this and we will refer to them as non-animation intervals, intervals that only return back duration, and animation intervals, intervals that return back the additional animation metrics. With Xcode 11, you could only instrument a non-animation os_signpost interval using the .begin and .end interfaces. This would just return back duration.
Now in Xcode 12, to specify an animation os_signpost interval, all you need to do is use the .animationBegin interface instead. With just this one change, you can convert any of your existing instrumentation to emit animation intervals instead and receive back the animation metrics mentioned earlier.
Aside from using a custom interval, you can use one of the predefined UIKit instrumented intervals for testing around navigation transitions and scrolling.
These are sub-metrics provided on the XCTOSSignpostMetric class. Let's take a look at an example of writing a test using one of these sub-metrics.
Here I have a performance XCTest that is going to launch my application, tap on the Meal Planner cell, and swipeUP on the food collection view to scroll down. In this test, I'm specifying that I want to measure the scrollDeceleration sub-metric. In the body of the measure block, I'm swiping up and now with Xcode 12 you can customize the velocity of the scroll.
This test looks good so far but there is an improvement we can make. Remember by default and measure block is run five times to collect performance measurements, meaning that in the current implementation, we are going to swipe up five times back-to-back and will most likely be swiping over different content in each iteration. To avoid this, we want to reset the application state between runs. We can do this by using XCTMeasureOptions to let our measure block know that we are going to manually stop or measurement collection.
Then pass this into our measure block, call stop measuring, and then reset our application state. Now that we have our test written, we want to run it.
But before we do so, we first want to modify some settings on our test scheme to eliminate their impact on our performance measurements. We first want to make sure we use a separate test scheme for our performance XCTests.
Then, we want to select the release build configuration and disable the debugger.
We also want to disable automatic screenshot collection and turn off code coverage.
And finally, we want to turn off all diagnostic options. These are the options listed under Runtimes Sanitization, Runtime API Checking, and Memory Management.
Now, we can run our performance XCTest and view our results in the report UI. In the dropdown we can see our new animation metrics. Let's select the hitch time ratio metric. We see that we collected measurements for five iterations. And we got an average of 1.2 milliseconds per second for our hitch time ratio. As a next step, we could set this average value of 1.2 milliseconds per second as our baseline so that any future runs of this test would be compared to this baseline value. Let's take a look at an example of how you might encounter a hitch in your code base and how you can use a performance XCTest to prevent it from shipping to your customers.
Let's say I'm an app developer at a Meal Planner company that wants to support online orders and deliveries. So far I've implemented a view that lists the different menu items available. As a next step, I want to make my food appealing by including images of what the different dishes look like. Before I dive into writing this new feature, I want to measure my current animation performance so I can use it as a baseline to compare for after I add the feature. I've set up a separate test scheme for my performance test and I've already configured the settings to what we talked about earlier.
I can now write my test. As we saw before, I'm going to launch my application, tap on the meal planner cell, and measure my scrolling animation performance. I've already pre-ran this test. So let's take a look at our measurements. It looks like we have zero hitches and our animations are performing as expected.
Let's go ahead and add in our new feature. First, I'm going to set my image view to contain the images I've already included in my project.
Second, I'm going to scale these images so that they fit well on my application.
Now we can go ahead and rerun our performance test. Note that when using the XCTOSSignpostMetric, the measure block will listen for when you're instrumented os_signpost interval is emitted within the code block and collect metrics only for the code executed within this interval. Another thing to note is that a measure block supports listening for multiple distinct os_signpost intervals. For example, you could listen for both the scrollDeceleration and scrollDragging os_signpost intervals within the same block of code that calls swipeUp(). Let's skip ahead to the completion of this test. It looks like we have an increase in the number of hitches and we should immediately investigate.
It looks like our issue is here in our scaleAspectFit call. We are re-drawing the image on the main thread which is responsible for rendering the rest of the UI. We are using the CPU which creates new pixels and allocates memory.
We can reduce this impact by using CoreAnimation’s set setContentMode which will hand off redrawing these images to the GPU. This allows us to use the existing image pixels reducing the amount of work we do on the main thread.
We can rerun our Performance XCTest again to see if this resolves the issue.
We can confirm that our animation metrics now report back zero hitches and our performance is back to what we expect. Using our Performance XCTest we were able to see that our new feature caused a regression, giving us a chance to fix it, and that now our feature is ready to reach our customers.
Let's recap what we've talked about. We learned that a hitch occurs any time a frame appears on the screen later than expected and we can quantify these hitches using the recommended good, warning, and critical categories.
We then learned that we can catch hitches in our development workflow using Performance XCTest. We can do this by using a UIKit or custom animation os_signpost interval. We also talked about best practices, which include resetting our application content between iterations and configuring our scheme settings to prevent inaccuracies. With this knowledge, you are now ready to prevent hitches in your codebase and provide smooth animation experiences to your customers. Thank you for listening and we hope you
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.