Optimize for variable refresh rate displays

Hi, my name is Kyle Sanner, from the GPU Software Engineering team. Together with my colleague Alex Li, we'll be talking about how to get optimal frame pacing in your apps on variable refresh-rate displays. We'll be focusing on some new display technology coming to macOS, Adaptive-Sync, and how to drive custom drawing smoothly on iPad Pro under all conditions. We'll start with a quick overview of the types of displays now supported on Apple Platforms.

We're going to introduce you to Adaptive-Sync displays on the Mac, and the new tools in macOS Monterey that you can use to deliver smooth frame rates in full-screen apps and games on these displays.

Then, we'll dig deep into ProMotion on iPad Pro, and look at some CADisplayLink best practices that help your apps maintain correct frame pacing at different frame rates.

Let's first review the types of displays Apple's devices can support.

Most displays on Apple systems work at fixed refresh rates. That is, they refresh themselves at a consistent rate whenever they are powered on. The exception is our ProMotion displays on iPad, and now with Adaptive-Sync displays on macOS. Let's start digging in to what's new with Adaptive-Sync displays on the Mac.

We'll start with what an Adaptive-Sync display is, and how they work on the Mac. But first, let's first take a quick look at how a fixed-rate display works.

Here's a diagram showing frames being delivered to a 60Hz display. Each frame is presented to the display, where it stays for 16ms before the display is refreshed. If there's a new frame drawn by your Mac ready in the Framebuffer, then that new frame is presented. Otherwise, the previous frame is shown again.

Looking at a 120Hz display, you see that that though we've doubled the refresh rate, and thereby halved the interval each frame can be onscreen, it behaves the same way, just faster.

Take a look at this Adaptive-Sync display, on the other hand. Instead of a static duration, each frame has a window of time that it can be onscreen. This window varies depending on the attached display. This display can operate between 40 and 120Hz, which means that a frame can be onscreen anywhere between 8 and 25ms.

Note that once the maximum time has elapsed, the system must refresh the panel, and the display will briefly be unavailable for new updates for a short time. Okay, so what kind of benefits can your games and apps get on an Adaptive-Sync Display? For applications that mostly run at the maximum refresh rate of the display, Adaptive-Sync displays provide a great benefit for free. Let's first take a look at this scenario, where your app is mostly able to produce new frames in under 8ms, so you're running fairly reliably at 120Hz. But due to a momentary increase in scene complexity, the finished frame lands in the Framebuffer 9ms after the previous frame was first displayed. On a fixed frame rate display, the previous frame is displayed for 16ms, instead of the 8ms you intended. This results in a perceptible hitch in your app.

On an Adaptive-Sync display, your frame is presented to the display immediately after it is done, so your app incurs only a 1ms penalty. Hitches this small are generally not perceptible to users. For workloads that can't reach the maximum frame rate of the display, you can provide smooth, even frames by making some small changes to how your app presents its drawables. Consider this scenario: a game running a complex scene can produce updates at around 90Hz. However, an intermittent effect causes a large jump in complexity, but does so inconsistently, causing sudden spikes down to 66Hz. By monitoring your app's GPU work, you can respond to this spike in complexity by intentionally presenting your frames later until your scene's complexity is consistently lower. Now, let's talk about some Adaptive-Sync best practices. On a fixed rate display, when your app's GPU work consistently exceeds the display's on-glass interval, we've previously recommended that you slow down your rendering to hit the next factor of the display's fastest refresh rate.

Typically, that means lowering your target frames per second to 30 from 60, like in this example here.

When presenting to an Adaptive-Sync display, however, we're changing that guidance. You should instead attempt to present frames at the highest rate your app can do so evenly. In addition to presenting frames evenly, remember that if your frames are presented at less than the minimum rate that the display supports, that the display might become unavailable for new frames, which could induce judder in your app. But so long as you're in the supported range, you're free to pick whatever rate works best for your app. Now that you've got a high-level understanding of the new display support coming to the Mac this year, let's talk about how to enable Adaptive Sync in your games.

First, you'll need a supported Mac. Any Mac with an Apple Silicon GPU will work great, and we also support many of our most recent Intel-based Macs as well. Second, you'll need a supported Adaptive-Sync display, and to enable Adaptive-Sync mode. This can be done by selecting the new variable refresh rate available in Display System Preferences. And lastly, your app needs to be running in full-screen mode. Let's see what APIs you can call in your app to detect whether it should be attempting to do Adaptive-Sync scheduling. First, you'll need to determine whether or not the display you're running on is capable of Adaptive-Sync scheduling. For that, there are some new properties this year on NSScreen, minimumRefreshInterval and maximumRefreshInterval. These values tell you the range of valid onscreen times for a frame presented to this display. On a fixed-frame display, these values will be the same, so a simple not-equal comparison will tell you whether this screen is in Adaptive-Sync mode. Next, you'll need to know if your window is currently full screen. You can determine that by checking your window's styleMask.

And remember, you'll need to combine both of these checks to ensure your app is able to take advantage of Adaptive-Sync scheduling.

Okay, great. So, now that you've got a handle on Adaptive-Sync displays and the new APIs that macOS provides to detect them, let's see how we can adapt some existing metal presentation techniques to present evenly on an Adaptive-Sync display.

You can use our MetalDrawable APIs that have built-in frame-pacing, such as presentAfterMinimumDuration or presentAtTime, to great effect with Adaptive-Sync displays. Or, you can roll your own solution with a present now call and your own custom timers. Let's take a look at how a few different implementations will work.

We'll start with a simple example. Here, we're going to acquire a Drawable, set up our GPU work, and present it on the screen. We're relying on the back pressure of a Drawable being available to set our frame rate for us. On a fixed-rate display, we know that this isn't the best idea, since there's no guarantee that your GPU work will align to the refresh rate of the display.

But as you can see from this instruments capture taken on an Adaptive-Sync display, when our scene is consistent, this seems to work out okay. The problem here is that this scene is running into periodic hitches. These hitches will translate into stutters that are visible to users. Let's try to fix that by presenting at a fixed, even rate. This technique can also be used if you want to implement a user-adjustable FPS slider for the players of your game. Here, we've set the frequency we want to 78Hz. And instead of a plain present call, we'll use present afterMinimumDuration for this Drawable, and specify the interval that we defined above. And here, you can see smoothly presented frames at the rate that we requested. We aren't presenting as quickly as the previous example, but your users are far less likely to encounter stutters, and your app will use less CPU and GPU time. Okay, so here's where things get a little more interesting. Let's try an approach that will produce evenly-paced frames without having to set a single fixed rate. One way to do this is to compute a rolling average of the GPU work needed to produce each frame, and feed that time into our present Drawable call. For the first frame, we need to load our average GPU time with a starting value. I'm going to choose to be optimistic and target the fastest rate the display can support here. This will just be a starting point for our average, so any reasonable guess we make is fine. Now, let's attach a CommandBuffer completion handler to measure the amount of time the GPU spent rendering this frame, and incorporate that time into our rolling average. First, we can acquire the time the GPU took to complete our work. Then, we'll incorporate that new time into our rolling average that will be used when we present the next frame. And here are the results. As you can see, we're presenting at a rate similar to the previous example, but this limit is determined by the previous frames we've generated, and will produce even frame rates across a range of Mac GPUs. Here, we can see the same program running smoothly at 48Hz on a less powerful Mac, without any additional code changes.

All right, Now you've got some new tools and techniques that you can use to optimize your app for Adaptive-Sync displays. If you want to learn more about Adaptive-Sync displays on macOS, check out the new Metal sample project on the Apple Developer site. To learn more about delivering performant experiences in Metal, check out these WWDC talks from previous years. And now, I'll hand you over to Alex, where you'll learn more about frame pacing on iPad Pro. Thank you, Kyle. Next, let's talk about ProMotion. Since 2017, every iPad Pro has been equipped with a ProMotion display that delivers refresh rates of up to 120Hz. However, 120Hz may not be available in some situations, including when the user has switched on Low Power Mode, which has been brought to the iPad this year with iPadOS 15. Proper frame pacing will allow your app to present motion contents correctly and smoothly, regardless of display characteristics, user preferences, and system states. We are going to look at the differences between ProMotion and fixed rate displays, as well as the situations in which some frame rates may not be available. Next, we'll discuss what is a display link, and how your app can use it to drive custom drawing. And finally, we'll offer some display link best practices. Let's dive right in. As Kyle has briefly presented earlier, a fixed 60Hz display refreshes every 16ms, a fixed cadence. It supports smooth presentation of contents whose frame rates are factors of 60. For example, 60Hz, 30Hz, 20Hz, and so on. However, when the content is slower than the display refresh rate, say 30Hz, the display itself still has to be refreshed at the same cadence, hence, every other frame is a repeat of the previous, and this consumes some power. On the other hand, ProMotion offers great responsiveness with refresh rates of up to 120Hz. It also adapts to onscreen content and so reduces its power consumption. Let's see how it works. Of course, at its maximum refresh rate of 120Hz, the display refreshes every 8ms.

Since 120 is a multiple of 60, ProMotion supports all existing frame rates. It offers not just 120Hz, but some intermediate frame rates for your apps as well. Moreover, ProMotion can dynamically adjust its refresh rate, so with a smooth 60Hz content, it can refresh only every 16ms without repeats, which otherwise would be required on a fixed 120Hz display. This is true all the way down to 24Hz.

Now, these frame rates may not always be available. The user can turn on Limit Frame Rate toggle in Accessibility settings that caps the maximum frame rate to 60Hz. Also, when the device gets hot, the system may apply restrictions on the availability of 120Hz. With iPadOS 15, we'll also enforce the 60Hz cap in Low Power Mode. So, how do these scenarios affect your apps? The good news is that most apps will work without any changes. But if your app performs frame-by-frame custom drawing, then you would need to pay attention to these frame rate changes, and we will show you how to do that. The recommended tool to drive custom drawing is display link, which is essentially a timer that is synchronized with display refresh rate. It helps your app drive any custom animations or custom render loop. There are two display links. One is CVDisplayLink, offered by CoreVideo on macOS, and the other is CADisplayLink, offered by CoreAnimation on our other platforms, as well as Catalyst on macOS, each with slightly different characteristics and behavior. Today, we'll only discuss CADisplayLink, but on a high level, these concepts will apply to both.

CADisplayLink wakes up at every vsync and invokes the callback. This provides the application the entire 8ms to complete its work.

A regular timer, such as an NSTimer, is very unlikely to be in perfect sync with the display. It can be out of phase or drifting, so sometimes the app may not have enough time to complete its work and it leads to frame drops. Now you've seen how CADisplayLink provides consistent timings, here are some of its additional benefits. It can run at a slower rate than the display refresh rate, and to do so, your app provides a hint via preferredFramesPerSecond and we will choose the nearest available frame rate for you. When the frame rate availability changes, as we have discussed earlier, CADisplayLink will automatically adjust its rate under the hood. Of course, it also provides your app with the necessary timing information so that your custom drawing can be aware of these changes. We won't go into how to write a custom animation or custom render loop, but we will provide you with four best practices to help your custom drawing stay in sync with display timings and avoid some of the common pitfalls.

First, it is important to query the display refresh rate at runtime instead of hard-coding it. Second, it is usually the case that you should use the frame rate of the CADisplayLink itself. Next, using targetTimestamp to prepare the drawing will help reduce hitches. Finally, it is always a good idea to prepare for the unexpected by dynamically computing the time delta. Let's go through them one by one. The maximum display refresh rate can be queried via UIScreen, which will always return 120Hz on ProMotion displays, even during situations such as when Low Power Mode is turned on. On the other hand, CADisplayLink will actually provide the shortest interval between frames via the duration property, and it will dynamically update based on the current device state. But almost always, you should use the actual frame information directly from the CADisplayLink because the display link can run slower than the maximum display refresh rate. Also, frame rate availability is dependent on the hardware, and the actual frame rate may be changed dynamically by the display link itself in response to system state changes. Let's look at an example. Suppose we request a 40Hz display link. As you see, on a ProMotion display, 40Hz is supported. However, on a 60Hz display, or when ProMotion is capped at 60Hz, the display link will automatically adjust itself to 30Hz. This ensures a good cadence where each wake-up is on a possible vsync that tries to give the equal amount of time for each frame. If we were to use a plain 40Hz NSTimer, which is not frame rate aware, its wake-up could be right in the middle of the vsync interval, and of course, we cannot present a frame there, so you'll likely observe hitches in your custom drawing. So, how does it look in the code? Well, here is how you would usually set up a display link. First, you must provide a target and a selector, which is the callback to be invoked. Next, hint the preferred frame rate of 40Hz via preferredFramesPerSecond. You then add the display link to the current runloop, from which the callback will be invoked. So, in the callback, you can get the expected interval between display link wake-ups by subtracting the timestamp from targetTimestamp. This interval is not necessarily always 1 over 40 because the display link itself may be running at a different frequency. Next, let's talk about these timestamps. There are primarily two timestamps on the CADisplayLink, Timestamp, which denotes when the callback is scheduled to be invoked, and targetTimestamp, which is when the next frame will be composited by CoreAnimation. We'll walk through an example that illustrates why you should use targetTimestamp to prepare your drawings. Here's an animation in its normalized time domain from 0 to 1. Suppose we are targeting the highest frame rate possible and currently it is 120Hz. CADisplayLink wakes up, and if we were to prepare our frame presentation using timestamp, we'll sample directly here, which gets presented in the next vsync, and here it is.

The same process continues, and we see that it has a good cadence where for each 120Hz frame, our animation progress increases by 0.05. Now, suppose the thermal state changes, and 120Hz is no longer available. Now the display link wakes up again, and the app prepares the animation at progress 0.4, which gets presented in the next vsync right here.

And the same pattern continues. Something's not quite right at the transition here. We see that the progress increases by 0.05, but one is over 8ms, and the other is over 16ms. It's very clear if we plot progress versus time, we'll see a hiccup right at the transition, and this will reflect as a user perceptible hitch, and that's not desirable.

Now, let's try targetTimestamp. CADisplayLink wakes up here. The progress is sampled at targetTimestamp, which gives 0.15. The same pattern continues, and again, we see a good cadence. At this frame rate transition point, the display link wakes up, samples at targetTimestamp, and we get 0.50. And it continues in the same way. If we plot the same progress versus time graph, you will see that it is a straight line, and hence it provides smooth contents even when the frame rate changes. So, targetTimestamp should be used rather than timestamp to prepare your drawings. In your code, it should generally be as simple as replacing any timestamp usage with targetTimestamp. Finally, let's talk about dynamically computing the time delta. The difference between targetTimestamp and timestamp gives you the expected amount of time between display link callbacks, but the actual amount of time is not guaranteed. A higher priority thread may be scheduled on the CPU, or the runloop is busy with something else. In the extreme case, callbacks may be skipped entirely, so in these situations, it's especially critical to still maintain the correct timing in your custom drawing for the best user experience.

When the CADisplayLink callback is invoked, the app performs its work to prepare the updates or renders necessary for the next frame. Usually, the callback will be invoked right at the scheduled wake-up time, but it's not alays the case. We expect the next callback to be invoked here. However, the display link doesn't get a chance to run until a few milliseconds into the vsync interval. And hence, you may not get the full 8ms. In this case, you can query CACurrentMediaTime and compare with targetTimestamp to get a sense of how much time is available.

Now, suppose the work is taking too long in this frame. The next callback won't be invoked until the runloop is free again. Because this one is delayed, the following callback will be skipped, so when you are preparing to advance the progress of your custom drawing in this callback, be mindful that the time delta that you should use is not 8ms, but rather 16ms, if you were to keep track of the previous timestamp at which your custom drawing state was updated at. Therefore, if your app uses time delta to advance the state of your custom drawing, this will slow down your custom drawing by one frame every time a calback is skipped. You can instead keep track of a previous targetTimestamp so that you can advance the state correctly. And if your custom drawing has high workload, you can look at targetTimestamp to potentially reduce the workload to meet the deadline as needed.

To recap the best practices, don't guess the display refresh rate. Always query it at runtime. Your custom drawing should be flexible in its supported frame rates and should be ready to adapt to a different rate. Use targetTimestamp to ensure a hitch-free frame rate transition, and be on the look out for any unexpected situations, such as missed display link callback. So, let's wrap up. In the first half of this session, we've discussed how to optimize your app's frame-pacing when running on an Adaptive-Sync display on macOS. In the second half, we've described the best practices for your app to drive its custom drawing and maintain smooth frame-pacing under all conditions on a ProMotion display on iPad Pro. As display technologies continue to evolve, we hope this session has provided you with not only insights, but also tools and best practices to support the increasingly dynamic timings of the displays. Thank you so much for joining us, and enjoy the rest of WWDC 2021. [music]