Image Filters

Apply high-performance filters to, and extract statistical and histogram data from images.

Overview

The MPSUnaryImageKernel and MPSBinaryImageKernel base classes define several properties common to all image kernels:

clipRect and clipRect

A clip rectangle is available to all image kernels that write to a destination texture. It describes the sub-rectangle of the destination texture overwritten by the filter. If the clip rectangle is larger than the destination texture, then the intersection between the clip rectangle and the destination texture bounds is used instead. A clip rectangle may be used to avoid doing work to obscured regions of the destination image, or to manage tiling and limit operations to parts of an image—for example, if a user draws a rectangle on the screen and asks your app to just apply the filter there.

offset, primaryOffset, and secondaryOffset

An offset is available to all image kernels that use a source texture from which pixel data is read. It describes the positioning of the source image relative to the result texture. An offset of {0, 0, 0} indicates that the top left pixel of the source texture is the center pixel used to create the top left corner of the destination texture clip rectangle (as a further example, an offset of {1, 2, 0} positions the top left corner of the clip rectangle at position x=1, y=2, and z=0 of the source image). The offset is the position of the top left corner of the clip rectangle in the source coordinate frame. It can be used for tiling and for translating an image up, down, left, or right by pixel increments. If there is no clip rectangle, then the offset is the top left corner of the region read by the filter. If there are multiple source textures, then the primary offset describes the top left corner of the region read in the primary source texture and the secondary offset describes the top left corner of the region read in the secondary source texture.

edgeMode, primaryEdgeMode, and secondaryEdgeMode

An edge mode describes the behavior of texture reads that stray off the edge of the source image. This can happen if the offset is negative, meaning a read off the top or left edge of the image. This can also happen if the sum of the clip rectangle size and the offset is larger than the source image, meaning a read off the bottom or right edge of the image. Furthermore, it is also possible for image filters to have a kernel window that stretches to examine neighboring pixels beyond the image bounds (such as convolution, morphology, and resampling filters). If there are multiple source textures, then the primary edge mode describes the mode to use with the primary source texture and the secondary edge mode describes the mode to use with the secondary source texture.

In-Place Operation

Some kernels can operate in place. This means that the same texture is used to hold both the input image and the result image. Operating in place is a great way to save memory, time, and energy. You can perform an in-place operation by using the encodeToCommandBuffer:inPlaceTexture:fallbackCopyAllocator: method.

Unfortunately, it is not always possible for kernels to run in place. Whether a particular kernel can operate in place can vary according to the hardware it is running on, the OS version, and the parameters and properties passed to it. You may not assume that because a kernel works in place today on a particular device that it will do so in the future.

To simplify error handling with failed in-place operation, the encodeToCommandBuffer:inPlaceTexture:fallbackCopyAllocator: method takes an optional MPSCopyAllocator object. It is used to create a new texture when in-place operation is not possible so as to allow the operation to proceed out of place in a reliable fashion instead. When this happens, the input texture is released and replaced with a new texture. To make use of the feature, you will need to write a copy allocator block.

Listing 1 shows a minimal copy allocator implementation. For more information, see the MPSCopyAllocator reference.

Listing 1

Minimal MPSCopyAllocator Implementation

MPSCopyAllocator myAllocator = ^id <MTLTexture>(MPSKernel * __nonnull filter, __nonnull id <MTLCommandBuffer> cmdBuf, __nonnull id <MTLTexture> sourceTexture)
{
    MTLPixelFormat format = sourceTexture.pixelFormat;
    MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat: format width: sourceTexture.width height: sourceTexture.height mipmapped: NO];
 
    id <MTLTexture> result = [cmdBuf.device newTextureWithDescriptor: d];
 
    return result;
    // d is autoreleased.
};

Supported Pixel Formats for Image Kernels

All Metal Performance Shaders image kernels support source and destination textures with the following ordinary and packed pixel formats:

MTLPixelFormatR8Unorm, MTLPixelFormatR8Unorm_sRGB

Ordinary formats with one 8-bit normalized unsigned integer component.

MTLPixelFormatRG8Unorm, MTLPixelFormatRG8Unorm_sRGB

Ordinary formats with two 8-bit normalized unsigned integer components.

MTLPixelFormatRGBA8Unorm, MTLPixelFormatRGBA8Unorm_sRGB, MTLPixelFormatBGRA8Unorm, MTLPixelFormatBGRA8Unorm_sRGB

Ordinary formats with four 8-bit normalized unsigned integer components.

MTLPixelFormatR16Float, MTLPixelFormatRG16Float, MTLPixelFormatRGBA16Float

Ordinary format with 16-bit floating-point components.

MTLPixelFormatR32Float, MTLPixelFormatRG32Float, MTLPixelFormatRGBA32Float

Ordinary format with 32-bit floating-point components.

MTLPixelFormatR16Unorm, MTLPixelFormatRG16Unorm, MTLPixelFormatRGBA16Unorm

Ordinary format with 16-bit normalized unsigned integer components.

MTLPixelFormatB5G6R5Unorm, MTLPixelFormatA1BGR5Unorm, MTLPixelFormatABGR4Unorm, MTLPixelFormatBGR5A1Unorm

Packed 16-bit format with normalized unsigned integer color components.

MTLPixelFormatRGB10A2Unorm

Packed 32-bit format with normalized unsigned integer color components.

MTLPixelFormatRG11B10Float, MTLPixelFormatRGB9E5Float

Packed 32-bit format with floating-point color components.

Some compressed pixel formats can be used as source textures. They cannot be used as destination textures because they cannot be written to. Metal Performance Shaders image kernels support the following compression families:

  • PVRTC

  • EAC/ETC

  • ASTC

The following Metal Performance Shaders image kernels also support source and destination textures with ordinary signed and unsigned integer pixel formats:

The ordinary signed and unsigned integer pixel formats supported by these image kernels:

MTLPixelFormatR8Sint, MTLPixelFormatRG8Sint, MTLPixelFormatRGBA8Sint

Ordinary format with 8-bit signed integer components.

MTLPixelFormatR8Uint, MTLPixelFormatRG8Uint, MTLPixelFormatRGBA8Uint

Ordinary format with 8-bit unsigned integer components.

MTLPixelFormatR16Sint, MTLPixelFormatRG16Sint, MTLPixelFormatRGBA16Sint

Ordinary format with 16-bit signed integer components.

MTLPixelFormatR16Uint, MTLPixelFormatRG16Uint, MTLPixelFormatRGBA16Uint

Ordinary format with 16-bit unsigned integer components.

MTLPixelFormatR32Sint, MTLPixelFormatRG32Sint, MTLPixelFormatRGBA32Sint

Ordinary format with 32-bit signed integer components.

MTLPixelFormatR32Uint, MTLPixelFormatRG32Uint, MTLPixelFormatRGBA32Uint

Ordinary format four 32-bit unsigned integer components.

For more information on pixel formats, see MTLPixelFormat and Pixel Format Capabilities.

Sample Code

Listing 2

Metal Performance Shaders Sample Code

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>
 
// Blur the input texture (in place if possible) on MTLCommandQueue q, and return the new texture.
// This is a trivial example. It is not necessary or necessarily advised to enqueue a MPSKernel on
// its own MTLCommandBuffer or using its own MTLComputeCommandEncoder. Group work together.
 
// Here we assume that you have already gotten a MTLDevice using MTLCreateSystemDefaultDevice() or
// MTLCopyAllDevices(), used it to create a MTLCommandQueue with MTLDevice.newCommandQueue, and
// similarly made textures with the device as needed.
void  MyBlurTextureInPlace(id <MTLTexture> __strong *inTexture, float blurRadius, id <MTLCommandQueue> q)
{
    // Create the usual Metal objects.
    // MPS does not need a dedicated MTLCommandBuffer or MTLComputeCommandEncoder.
    // This is a trivial example. You should reuse the MTL objects you already have, if you have them.
    id <MTLDevice> device = q.device;
    id <MTLCommandBuffer> buffer = [q commandBuffer];
 
    // Create a MPS filter.
    MPSImageGaussianBlur *blur = [[MPSImageGaussianBlur alloc] initWithDevice: device];
    if( nil == blur )
        MyHandleError(kOutOfMemory);
 
    blur.sigma = blurRadius;
    // Defaults are okay here for other MPSKernel properties (clipRect, origin, edgeMode).
 
    // Attempt to do the work in place.  Since we provided a copyAllocator as an out-of-place
    // fallback, we don’t need to check to see if it succeeded or not.
    // See the "Minimal MPSCopyAllocator Implementation" code listing for a sample myAllocator.
    [blur encodeToCommandBuffer: commandBuffer inPlaceTexture: inTexture copyAllocator: myAllocator];
    [blur release];
 
    // The usual Metal enqueue process.
    [buffer waitUntilCompleted];
 
    return result;
}

Topics

Morphological Image Filters

MPSImageAreaMax

A filter that finds the maximum pixel value in a rectangular region centered around each pixel in the source image.

MPSImageDilate

A filter that finds the maximum pixel value in a rectangular region centered around each pixel in the source image.

MPSImageAreaMin

A filter that finds the minimum pixel value in a rectangular region centered around each pixel in the source image.

MPSImageErode

A filter that finds the minimum pixel value in a rectangular region centered around each pixel in the source image.

Convolution Image Filters

MPSImageConvolution

A filter that convolves an image with a given kernel of odd width and height.

MPSImageMedian

A filter that applies a median filter in a square region centered around each pixel in the source image.

MPSImageBox

A filter that convolves an image with a given kernel of odd width and height.

MPSImageTent

A filter that convolves an image with a tent filter.

MPSImageGaussianBlur

A filter that convolves an image with a Gaussian blur of a given sigma in both the x and y directions.

MPSImageGaussianPyramid

A filter that convolves an image with a Gaussian pyramid.

MPSImageSobel

A filter that convolves an image with the Sobel operator.

MPSImageLaplacian

An optimized Laplacian filter, provided for ease of use.

MPSImageLaplacianPyramid

A filter that convolves an image with a Laplacian filter.

MPSImageLaplacianPyramidAdd

A filter that convolves an image with an additive Laplacian pyramid.

MPSImageLaplacianPyramidSubtract

A filter that convolves an image with a subtractive Laplacian pyramid.

MPSImagePyramid

A base class for creating different kinds of pyramid images.

Histogram Image Filters

MPSImageHistogram

A filter that computes the histogram of an image.

MPSImageHistogramEqualization

A filter that equalizes the histogram of an image.

MPSImageHistogramSpecification

A filter that performs a histogram specification operation on an image.

Image Threshold Filters

MPSImageThresholdBinary

A filter that returns a specified value for each pixel with a value greater than a specified threshold or 0 otherwise.

MPSImageThresholdBinaryInverse

A filter that returns 0 for each pixel with a value greater than a specified threshold or a specified value otherwise.

MPSImageThresholdToZero

A filter that returns the original value for each pixel with a value greater than a specified threshold or 0 otherwise.

MPSImageThresholdToZeroInverse

A filter that returns 0 for each pixel with a value greater than a specified threshold or the original value otherwise.

MPSImageThresholdTruncate

A filter that clamps the return value to an upper specified value.

Image Integral Filters

MPSImageIntegral

A filter that calculates the sum of pixels over a specified region in an image.

MPSImageIntegralOfSquares

A filter that calculates the sum of squared pixels over a specified region in an image.

Image Manipulation Filters

MPSImageConversion

A filter that performs a conversion of color space, alpha, or pixel format.

MPSImageScale

A filter that resizes and changes the aspect ratio of an image.

MPSImageLanczosScale

A filter that resizes and changes the aspect ratio of an image using Lanczos resampling.

MPSImageBilinearScale

A filter that resizes and changes the aspect ratio of an image using Bilinear resampling.

MPSImageTranspose

A filter that transposes an image.

Image Statistics Filters

MPSImageStatisticsMean

A kernel that computes the mean for a given region of an image.

MPSImageStatisticsMeanAndVariance

A kernel that computes the mean and variance for a given region of an image.

MPSImageStatisticsMinAndMax

A kernel that computes the minimum and maximum pixel values for a given region of an image.

Image Reduction Filters

MPSImageReduceRowMax

A filter that returns the maximum value for each row in an image.

MPSImageReduceRowMin

A filter that returns the minimum value for each row in an image.

MPSImageReduceRowSum

A filter that returns the sum of all values for a row in an image.

MPSImageReduceRowMean

A filter that returns the mean value for each row in an image.

MPSImageReduceColumnMax

A filter that returns the maximum value for each column in an image.

MPSImageReduceColumnMin

A filter that returns the minimum value for each column in an image.

MPSImageReduceColumnSum

A filter that returns the sum of all values for a column in an image.

MPSImageReduceColumnMean

A filter that returns the mean value for each column in an image.

MPSImageReduceUnary

The base class for reduction filters that take a single source as input.

Image Arithmetic Filters

MPSImageAdd

A filter that returns the element-wise sum of its two input images.

MPSImageSubtract

A filter that returns the element-wise difference of its two input images.

MPSImageMultiply

A filter that returns the element-wise product of its two input images.

MPSImageDivide

A filter that returns the element-wise quotient of its two input images.

MPSImageArithmetic

Base class for basic arithmetic nodes

Euclidean Distance Transform Filter

MPSImageEuclideanDistanceTransform

A filter that performs a Euclidean distance transform on an image.

Fast Guided Filter

MPSImageGuidedFilter

A filter that performs edge-aware filtering on an image.

Keypoints

MPSImageFindKeypoints

A kernel that is used to find a list of keypoints.

MPSImageKeypointData

A structure that specifies keypoint information.

MPSImageKeypointRangeInfo

A structure that specifies information to find the keypoints in an image.

Image Filter Base Classes

MPSUnaryImageKernel

A kernel that consumes one texture and produces one texture.

MPSBinaryImageKernel

A kernel that consumes two textures and produces one texture.

Constants

MPSRectNoClip

The default clipping rectangle for a kernel object.