Boost performance with MetalFX Upscaling

♪ ♪ Hello and welcome. My name is Kelvin Chiu from the GPU Software team here at Apple. Today, I'll talk about how to boost your Metal application performance with MetalFX Upscaling.

MetalFX is a new API that provides platform optimized graphics effects for Metal applications.

It enables high performance upscaling, which will boost your application performance while retaining its rendering quality. Rendering a frame at high resolution costs GPU time. To reduce that time, rendering at a lower resolution will usually do the trick. However, the tradeoff is a lower rendering quality. With MetalFX Upscaling, your application can now render frames at a lower resolution, reducing rendering time, without compromising rendering quality. MetalFX Upscaling is optimized to run best on Apple devices. and it is also easy to adopt in your game. MetalFX provides two upscaling effects, which I will describe in detail. Spatial upscaling is simple to use and gives a great performance boost.

Temporal anti-aliasing and upscaling integrates information from multiple frames to produce a higher quality output.

I will then talk about best practices for using these effects. Finally, I will end the session with demos showing them in action.

Let's start with Spatial upscaling.

MetalFX Spatial upscaling analyzes the input’s spatial information to produce new, upscaled samples. Integrating Spatial upscaling is simple. It only requires anti-aliased input color to produce a spatially upscaled color output. In a typical game rendering pipeline, there are various rendering passes including anti-aliased render and various post processing effects. Add MetalFX spatial upscaling right after the game's tone-mapping process is completed. It will perform best if the input has been tone mapped and is in a perceptual color space. Let’s checkout MetalFX spatial upscaling in action.

This chess scene is produced with a high quality reference renderer in 4K resolution. It is path traced, with complex graphics effects, like ray-traced reflections and shadows. Here is a side by side comparison, with 540p input on the left...

…and 1080p output with MetalFX spatial upscaling on the right.

If I zoom in on the queen, on the left, the image lacks details and is low resolution. On the right, the spatially upscaled output has sharper reflections and more refined edges.

Next, I'll walk you through how to implement MetalFX spatial upscaling.

In Metal, you would normally create a command encoder to encode commands into a command buffer and produce input for the effect. Similarly, you can create a MetalFX effect object to encode commands into a command buffer and perform the effect. Finally, create a third command encoder to encode commands that make use of the MetalFX output. You should only create a new spatial scaler object when your application first starts or when a display switches resolutions because they are expensive to create.

First, create and configure a MTLFXSpatialScalerDescriptor. Then, create a scaler object by calling the makeSpatialScaler() method. In the initialization code, start with the descriptor. Fill both the width and height of the input and output texture. Then, set the texture format for the textures that will be set later on the scaler object. Set the color processing mode. This tells the API which color space the input and output is in. You can set the value to be in either perceptual, linear, or HDR color space. Once the descriptor is filled, create the scaler object.

Once the scaler object is created, you can modify the properties of the object as frequently as you want and call the encode() method to start the upscaling process.

In your per frame draw code, make sure the correct input and output textures are being set on the scaler object before encoding the scaling effect into the command buffer.

Spatial upscaling offers a simple way to boost performance.

And when you want even higher quality rendering, that's where MetalFX temporal anti-aliasing and upscaling comes in. Temporal AA and upscaling is a technique that uses data from previous frames to produce high-quality upscaled output. This means, the output of upscaling from the previous frame will be used as one of the inputs for the current frame upscaling.

To better understand why Temporal AA and upscaling requires data from the previous frames, I'll first review the concept of supersampling.

In supersampling, multiple samples are calculated per pixel, which is then integrated into a single pixel value. The more samples that we integrate per pixel, the better the result will be. However, it comes at a great cost to calculate multiple samples per pixel within a single frame. Instead of sampling multiple locations per pixel in a single frame, you can perform temporal sampling. Temporal sampling is the concept of rendering a different sample location for all the pixels in a given frame. This enables you to achieve supersampling quality over multiple frames at a significantly lower cost.

By accumulating samples from multiple frames and accounting for sample positions, temporal AA & upscaling can integrate samples appropriately in target resolution pixels, resulting in a high quality anti-aliased upscaled output.

However, since content often changes between frames, it will require more input data to detect these changes.

Besides the previous frame output, Temporal AA and upscaling also requires a color input that is jittered, as well as motion and depth data from the scene.

I'll go through each of them to explain why they are required.

Let's start with the jittered color input.

Here is a red triangle rendered without jitter. The bright white outline represents the triangle being rendered.

Each one of the small squares represent a pixel. And the gray dot in the middle is where the pixel is sampled.

This is the output of the same triangle when rendered with a small jitter. The gray dots show the sampling location for a given pixel.

The jitter offset should be unique for a set number of frames in order to fully gather the desired number of samples.

I will cover the topic of jitter sequence in detail later.

Next is the motion information from the scene. Motion data from the scene indicates how much and which direction the objects had moved from the previous frame. Temporal AA and upscaling uses the motion information to back track and find corresponding locations in the previous frame in order to correctly gather samples.

Another input is the depth information from the scene.

Depth data from the scene indicates what's in the foreground and what's in the background. This is important when prioritizing foreground edge anti-aliasing and provides clues on what other objects might be newly exposed when gathering samples from previous frames. The last piece of input data is the previous frame’s output.

The previous frame’s output contains all of the samples that have been integrated previously, and it will be blended with the current frame’s jittered color input in order to increase the number of samples per pixel.

By combining information from both the current and previous frame, the resulting image now has more details. MetalFX keeps track of the upscaled output, so you only need to pass the color, motion, and depth from the current rendered frame.

Going back to a typical game’s rendering pipeline, MetalFX temporal AA and upscaling should run before any post processing effects, since these effects will interfere with the result of the upscaling.

Here's the chess rendering again, this time using MetalFX Temporal AA and upscaling. This is a side by side comparison of 1080p input on the left and 4K upscaled output on the right.

Zooming in closer to the queen, the input is low resolution and aliased, while the temporally upscaled output on the right is high resolution with a smooth outline and has more fine details in the reflections.

Just as with spatial scaler, creating a new temporal scaler is expensive and should only be done when your application first starts or when a display switches resolutions. First, you'll need to allocate and fill in a MTLFXTemporalScalerDescriptor.

Then call makeTemporalScaler() method to create the scaler object.

In your initialization code, start with the descriptor.

Fill in both the width and height of the input and output textures.

Then set the jittered color, depth, and motion texture formats for the textures that will be bound later on the scaler object as inputs.

Finally, set the format for the output texture where MetalFX will store the upscaled output. Once the descriptor is filled, create the scaler object.

On the scaler object, set the motion scale properties. This helps you scale the app's motion data to what the API expects.

MetalFX expects motion data in the render resolution pixel space with direction that goes from the current frame’s position to the previous frame’s position. As an example, I'll use a render resolution of 1080p. Suppose you have an object that moves from clip space coordinate (-0.75, -0.75). to clip space coordinate (0.25, 0.25). The motion data is stored as (1, 1).

Set the motion vector scale property to (-960, 540) so that MetalFX can interpret your game's motion data correctly.

You can modify the properties of the scaler object as frequently as you want. And call the encode() method to start the upscaling process.

For your per frame draw code, first set the resetHistory property. Set this to true when your application loads the first frame or when there is a scene cut. Then set the textures that are inputs for the effect, followed by the output texture. Next, set the reversedDepth property to indicate whether the depth values are in reversed-Z mapping.

The last property to set before encoding the scaling effect is the current jitter offset.

Getting jitter offset correctly is essential to the quality of the output. Let us take a quick look on how to set jitter offset.

As an example, on the left is a triangle rendered with jitter.

On the right is a zoomed in view of a pixel. The sample is located at (0.625, 0.78). The pixel center is represented by the orange dot. It is located at (0.5, 0.5).

In this example, the jitter offset is (-0.125, -0.28).

Note that jitter offsets are always in the range of -0.5 to 0.5. To verify that you are providing the correct jitter offset, render a scene without camera and object motion using a sequence of different jitter offsets. On the left is an example when incorrect jitter offset is specified. Static objects could shift, and fine lines are fuzzy.

On the right is the output when correct jitter offset is specified. Objects stay in place, and fine lines are resolved progressively. The MetalFX “temporal AA and upscaling” effect boosts your application performance and gives an upscaling quality that’s comparable to the quality of the native target resolution rendering. In order to get optimal quality and performance when using both upscaling effects, l’ll now cover implementation best practices.

Starting with spatial upscaling. For the best spatial upscaling quality, the color input should be anti-aliased and noise free. This is because noise effects and aliased images prevent good edge determination, which will worsen the spatial upscaling quality.

For the best performance, use the perceptual color processing mode. This means, your input color should be tone mapped, with values from 0-1, in the sRGB color space.

And finally, set the appropriate negative mip bias for higher texture detail. The recommended mip bias calculation for spatial upscaling is to apply log2 of the render resolution width, divided by the target resolution width.

For example, scaling each render resolution dimension by 2x will result in -1 mip bias, while scaling each dimension by 1.5x will result in -0.58 mip bias.

Note that lower mip levels might result in flickering for textures with high frequency patterns. You should adjust the mip bias for certain textures if you spot such artifacts. I will talk next about TemporalAA and upscaling best practices.

To get the best quality from Temporal AA and upscaling, it's important to choose a good jitter sequence. Look for a jitter sequence that will provide a good distribution of samples across all the pixels in an upscaled target resolution. Usually, eight jittered samples per output pixel produces a high-quality anti-aliased upscaled output.

In the case of 2x upscaling, the recommendation is to use a Halton (2,3) sequence with 32 jitters to produce your jittered color input. Here’s a plot of the first 32 sample locations from Halton (2,3) sequence, producing approximately eight samples per output pixel.

It's also important to set the appropriate negative mip bias for higher texture detail. The recommended mip bias calculation for temporal AA and upscaling is to apply log2 of render resolution width, divided by target resolution width, subtracted by 1.

For example, scaling each render resolution dimension by 2x will result in a -2 mip bias, while scaling each dimension by 1.5x will result in a -1.58 mip bias.

Next, I will show you examples of how mip bias affects your output in different situations.

Here are MetalFX temporal AA and upscaling outputs of the same scene using mip bias of 0, -1, and -2.

Mip bias of -2 produces the sharpest and clearest output, while mip bias of 0 produces the softest and most blurry output.

Here are three renderings of a circuit board that use the temporal upscaling effect. From top to bottom, the mip bias values applied when sampling textures are 0, -1, and -2. Because the circuit board’s textures have high-frequency patterns, such as its tiny trace wires, mip bias of -2 generates flickering and moire effects. However, mip bias of -1 greatly reduces these effects, and mip bias of 0 completely eliminates them.

Lower mip levels generally result in more details. Use our mip bias suggestion as a starting point, but be mindful when choosing a mip bias for textures with high-frequency patterns. Follow these practices to ensure an anti-aliased, high-quality upscaled output from MetalFX Temporal AA and upscaling.

Finally, I will cover performance best practice when using MetalFX Upscaling. To get the best performance with MetalFX Upscaling, you should be careful to avoid binding the same resources for reading and writing in two non-dependent render or compute passes. Doing so creates false dependencies. In Metal, it's always a good idea to avoid false dependencies. But this is especially important for MetalFX Upscaling, as I will describe next. In this example, there are two frames. The shadow and the post processing passes are completely unrelated and have no resource dependencies. Metal will overlap the next frame’s shadow pass with the current frame’s post processing pass.

However, if the post processing pass writes to a Metal buffer while the shadow pass also reads from the same buffer, Metal will prevent the GPU from running these two passes in parallel in order to avoid the potential hazard of reading and writing to the same resource at the same time. False dependencies between frames can negatively affect performance of MetalFX Upscaling. Notice that if there is no false dependency between frames, the next frame’s shadow pass could have overlapped with the previous frame's MetalFX Upscaling. However, because of the false dependency between frames, the performance loss now also includes the time it takes for MetalFX Upscaling to finish its process. Ideally, you should ensure that there are no false dependencies between frames to allow overlapping of workloads between different frames, ensuring the most optimal performance when using MetalFX Upscaling. In this example, you can instead create a separate buffer for the post processing and shadow passes to prevent the false dependency, resulting in parallel execution of independent passes.

Avoiding false dependencies is something you always want to keep in mind when adopting MetalFX Upscaling. When deciding which of these two effects to choose, there are some considerations you should also keep in mind.

With ever-increasing shading costs and pixel counts, temporal AA and upscaling is here to stay. Amortizing pixels temporally increases visual fidelity and boosts performance. If you don’t already have a great temporal AA solution and can render jittered color, motion, and depth buffers, MetalFX temporal AA and upscaling provides a compelling platform-optimized solution that you should consider. If you don’t have the necessary inputs or already have a well tuned AA solution, consider using MetalFX spatial upscaling. With that, hopefully you now have a better understanding of which upscaling effect to choose. I will next show examples of both of these effects running live in Metal applications. Here’s a side by side comparison of the “Bistro” scene from our “Modern Rendering with Metal” sample code, which features real-time rendering algorithms, like ambient occlusion and volumetric fog. Native rendering at 1080p on the left versus 4K output with MetalFX Spatial upscaling on the right. Note that this sample has its own temporal anti-aliasing solution, which we use as input for MetalFX spatial upscaling.

Zooming in more closely at the scooter...

On the left, the image is a bit blurry, while on the right, the spatially upscaled output results in a sharper image with cleaner edges. The straight line on the handle bar is nicely antialiased.

The curve on the body is also much smoother.

Let's do a performance comparison. On the left is a native rendering at 4K. On the right is the 4K output from MetalFX Spatial upscaling.

As the camera moves, the native rendering on the left is running at a choppy frame rate, while the spatially upscaled output on the right is much smoother.

Next is a side-by-side comparison of a ray-traced scene with many reflections and shadows.

On the left is a native rendering at 1080p. On the right is the 4K output with MetalFX Temporal AA and upscaling.

Zooming in more closely at the chandelier...

The native output on the left has an aliased look, while the temporally upscaled output on the right has sharp edges with more fine details. The shadow is nice and crisp rather than fuzzy looking. And fine details on the chandelier can now be recognized.

Performance gains are also apparent with MetalFX Temporal AA and upscaling. On the left is native rendering at 4K. On the right is the 4K output with MetalFX Temporal AA and upscaling. As the camera moves, the native rendering on the left is running at a very low frame rate, while the temporally upscaled output on the right is much smoother.

Leading game developers are excited by the capabilities of MetalFX Upscaling and will be bringing "Grid: Legends," "Resident Evil: Village," and "No Man’s Sky" to Mac later this year. Next, I’ll show you some early work using the framework. In this scene, we can see the incredible visuals and fluid gameplay of "No Man’s Sky" using MetalFX Temporal AA and Upscaling. To recap, MetalFX is a new API with a focus on upscaling. Spatial upscaling is easy to adopt and delivers substantial performance gains, and you can use Temporal AA and upscaling to deliver higher quality rendering. Following the best practices I talked about earlier will ensure you get the most out of MetalFX Upscaling. Thank you for watching. ♪ ♪