Explore Core Image kernel improvements

♪ Bass music playing ♪ David Hayward: Welcome, everyone.

My name is David Hayward.

I'm a senior engineer on the Core Image team, and I'll be giving a short presentation today that will show you the latest best practices when adding custom Metal Core Image kernels to your Xcode project.

In this presentation, I will discuss the general benefits of writing custom CIKernels in Metal.

Next, I will introduce two recommended ways that Metal CIKernels can be built.

And then I will demonstrate step by step how to add these to your project.

First off, let's review the benefits of writing custom CIKernels in Metal.

By writing CIKernels in Metal, you will get access to Core Image features such as automatic tiling and concatenation.

It will improve your app's performance by shifting some of the time to compile kernels from runtime to when your app is built.

And doing so will give your kernels access to high-performance features such as gather-reads, group-writes, and half-float math.

Last but not least, it will make your life as a developer easier by syntax highlighting as you type and inline error checking when you build.

So with that for motivation, I will now show you step by step how to add Metal Core Image kernels to your application.

There are now two recommended ways to add CIKernels to your project, and I will describe both methods in detail.

I will refer to this as the extern method because it requires the kernel functions to be specified as extern "C" and built using custom build flags.

The second method is new in iOS 15 and macOS 12.

I will refer to this as the stitchable method because it requires the kernel functions to be in attributed as stitchable.

Its implementation uses Metal Dynamic Libraries.

In both methods, there are four general steps to follow.

First, is to configure your project appropriately.

Second, is to add Metal CIKernel source files to your project.

Third, is to write your Metal CIKernel code.

And fourth is to write Swift or Objective-C code to initialize and apply your kernel to create a new CIImage.

First, let's describe these four steps when using extern CIKernels, starting with project configuration.

Unlike conventional Metal compute and graphics shaders, this Core Image Metal code needs to be compiled and linked with special flags.

I recommend adding two custom build rules to your project targets which will make using these flags automatic.

First, you will go to the project's target settings and add a build rule for files that end in .ci.metal.

For files with this extension, this rule will run a one-line script that calls the Metal compiler with the required -fcikernel flag.

This build rule will produce an output binary that will end in .ci.air.

Next, you will add a second build rule for files that end in .ci.air.

For files with this extension, this rule will run a one-line script that calls the Metal linker with the required -cikernel flag.

This build rule will produce an output in your app's Resources directory that will end in .ci.metallib.

Now that you have added the custom build rules, all you need to do is add .ci.metal sources to your project.

To do that, select from the File menu that you want to add a new Metal file and then give that new file a name ends in .ci.metal.

The next step is to write your CIKernel in the Metal source file.

First, at the top of the source, you will include the CoreImage.h header so that you get access to all the classes that Core Image provides.

The kernel must be specified as extern "C" to be recognized by Core Image.

The content of your actual kernel implementation is up to your imagination.

For one example of what you can do, I recommend that you watch our WWDC 2020 presentation on "HDR editing and playback using AVFoundation." The final step is to add Swift code to load your kernel and apply it to create a new image.

Kernels are typically used in a CIFilter subclass which will have properties such as an inputImage and other input parameters.

I recommend that your filter instantiate its CIKernel object into a static property.

This way, the work of loading the metallib resource is done only once when it is first needed.

Because of the custom build rule I described earlier, you will need to specify a resource URL with the same name as your source and an extension of .ci.metallib.

Lastly, a CIFilter subclass must override the outputImage property.

In the getter, you will take the kernel from a static property and use its apply method to create a new image based on the input properties.

So that fully describes the process of building extern CIKernels.

Let's now describe the new process of building stitchable CIKernels.

With the stitchable method, only one setting change is needed to configure your Xcode project.

This setting will tell the Metal linker to link against the Core Image framework.

To do this, just go to the project's target setting and add a build setting for Other Metal Linker Flags with the value "-framework CoreImage".

Because stitchable CIKernels don't require custom build rules, you can simply add .metal sources to your project without any special suffix.

You can add kernels in either one or several sources files.

By default, Xcode will build all of them into one .metallib resource.

The next step is to write your CIKernel in the Metal source file.

As before, you will include the CoreImage.h header to get access to the Core Image classes.

But with this method, the kernel must be attributed as ] to be recognized by Core Image.

Once again, the final step is to add Swift code to load your kernel, and apply it to create a new image.

The only change with stitchable CIKernels is that you can simply load the resource with the standard name of default.metallib.

That concludes the process of building stitchable CIKernels.

It is worth mentioning some of the benefits of using this method.

Stitchable kernels can link against other Metal libraries.

And they now support input parameters that are integer and unsigned integer vector types.

Another niche benefit is that stitchable kernels can be compiled from source at runtime.

Most applications should not use this feature because it will incur longer initial compile times.

That said, there are some classes of applications that may benefit from this flexibility.

There is one last topic to cover in this presentation.

The stitchable CIKernel implementation depends on two important Metal features.

The first is the new Metal Shading Language version 2.4.

Among other things, this versions supports the ] attribute that causes the compiler to associate additional metadata with each function.

The second is the Metal Dynamic Libraries feature that is used so that your kernels can link against the Core Image Metal classes.

For more details on these Metal features, be sure to watch the "Discover Compilation Workflows in Metal" presentation.

Be aware, though, that Metal Dynamic Libraries are only supported on some graphics devices, specifically iPhone and iPad with A11 and later, all Macs with Apple silicon, and Intel Macs with AMD Navi and Vega GPUs.

Your application should check the Metal device property supportsDynamicLibraries before using stitchable CIKernels.

So that concludes my step-by-step description of how to use the two recommended ways to add Metal CIKernels to your application.

For each method, I've covered how to configure your project, write the kernel source, and initialize the kernel objects.

I hope that this allows you to add great visual effects to your app's images and videos.

Thank you and enjoy the rest of WWDC 2021! ♪