Sample Code

Reducing Artifacts in Resampled Images

Avoid ringing effects introduced by the default Lanczos algorithm when scaling an image by using a custom resampling filter.

Download

Overview

In this sample code project, you’ll write a custom resampling filter to scale an image using linear interpolation.

Most of the vImage geometry operations, such as scale and rotate, use a process known as resampling to avoid image artifacts. vImage resamples with kernels that combine data from a target pixel and other nearby pixels to calculate a value for the destination pixel.

The following illustrates how resampling at fractional pixel locations scales a horizontal strip of pixels to three times its original width. For example, the scale operation sets the value of the leftmost pixel in the destination from position 0.333 in the leftmost source pixel.

Diagram showing how the pixels of a two pixel wide strip are sampled to generate a scaled copy that’s six pixels wide.

Because the resampling process has to evaluate the kernel at fractional pixel locations, the process relies on a family of kernel matrices for use at different fractional distances through a given pixel. You provide a function that generates this family of kernels. This is in contrast to operations such as convolution and morphology, for which you supply a single kernel matrix that’s applied at the center of each pixel.

Review Default Lanczos Resampling

For most vImage geometric operations, vImage supplies a default resampling filter that is an implementation of the Lanczos resampling method. However, the Lanczos method can produce ringing effects near regions of high frequency signals (that is, regions that contain a lot of pixel variation, such as the hard edges typical of line art). To correct this, you can implement linear interpolation as a custom resampling filter.

This sample app allows the user to toggle between the default resampling filter (Lanczos) and a custom resampling filter. Declare the filter independently of initialization to support that functionality:

let resamplingFilter: ResamplingFilter

let scale: Float = 30

The following code initializes a default Lanczos resampling filter:

resamplingFilter = vImageNewResamplingFilter(scale,
                                             vImage_Flags(kvImageHighQualityResampling))

On return, resamplingFilter is an initialized Lanczos resampling filter with the specified scale factor.

Use Shear Operations to Scale an Image

The vImage shear functions accept the resampling filter and perform the scaling. The shear functions operate in one dimension at a time, so to scale an image in both dimensions, call vImageVerticalShear_ARGB8888(_:_:_:_:_:_:_:_:_:) followed by vImageHorizontalShear_ARGB8888(_:_:_:_:_:_:_:_:_:). Because these functions require separate input and output buffers, use an intermediate buffer to pass data from the vertical shear to the horizontal shear.

var backColor = UInt8(0)

let height = Float(sourceBuffer.height)
let yTranslate = (height - height * scale) * 0.5
vImageVerticalShear_ARGB8888(&sourceBuffer,
                             &intermediateBuffer,
                             0, 0,
                             yTranslate,
                             0,
                             resamplingFilter,
                             &backColor,
                             vImage_Flags(kvImageNoFlags))

let width = Float(sourceBuffer.width)
let xTranslate = (width - width * scale) * 0.5
vImageHorizontalShear_ARGB8888(&intermediateBuffer,
                               &destinationBuffer,
                               0, 0,
                               xTranslate,
                               0,
                               resamplingFilter,
                               &backColor,
                               vImage_Flags(kvImageNoFlags))

On return of the horizontal shear, destinationBuffer contains the source image, scaled about its center. The following shows an original image, filled with small dots, magnified 30 times using the Lanczos resampling filter:

A scaled version of an image consisting of a field of small spots that shows ringing artifacts.

The ringing artifacts appear as faint lines between the magnified dots.

Write a Linear Resampling Filter Function

The shear functions used for scaling are both 1D and, therefore, the resampling filter function you create is also 1D. You can apply the same filter function for both the vertical and horizontal passes.

The function generates a set of kernel values based on a set of supplied distances from the pixel being transformed—read from inPointer. The generated kernel values are assigned to outPointer.

In the following example, the kernel values are inversely proportional to the distance; the further a pixel is from the transformed pixel, the smaller the corresponding kernel value. After calculating the kernel values, the values scale (normalize) so that so that their sum is 1. This normalization step ensures the final image is the same brightness as the original.

func kernelFunc(inPointer: UnsafePointer<Float>?,
                outPointer: UnsafeMutablePointer<Float>?,
                count: UInt,
                userData: UnsafeMutableRawPointer?) {
    if let inPointer = inPointer, let outPointer = outPointer {
        let absolutePixelPositions =
            Array(UnsafeBufferPointer(start: inPointer,
                                      count: Int(count))).map {
                                        abs($0)
            }

        let kernelValues = absolutePixelPositions.map {
            (absolutePixelPositions.max()! - $0)
        }
        
        let normalizedKernelValues = kernelValues.map {
            $0 / kernelValues.reduce(0, +)
        }
        
        outPointer.assign(from: normalizedKernelValues,
                          count: Int(count))
    }
}

The userData parameter allows you to optionally pass custom data to your resampling kernel function. It’s not used in this sample app.

For example, if the pixel positions passed to inPointer are:

[-3.0, -2.0, -1.0, 0.0, 1.0, 2.0, 3.0, 4.0]

The values in the kernelValues array are:

[1.0, 2.0, 3.0, 4.0, 3.0, 2.0, 1.0, 0.0]

Dividing each of the values in kernelValues by its sum returns the normalized kernel values that you assign to the resampling function’s outPointer:

[0.0625, 0.125, 0.1875, 0.25, 0.1875, 0.125, 0.0625, 0.0] // sum = 1

The values generated by the resampling function form a 1D convolution kernel that the shear functions use in a similar way to the 1D convolution described in the Blur an Image with a Separable Kernel section of Blurring an Image. However, unlike the kernels used for convolution, the resampling kernel is suitable for use with fractional pixel positions.

Allocate Resampling Filter Function Memory

The resampling function, the scale factor, and the kernel width combine to determine the memory required by the resampling function. Use the vImageGetResamplingFilterSize(_:_:_:_:) function to calculate the size in bytes, and the allocate(byteCount:alignment:) function to allocate the neccessary memory.

let kernelWidth: Float = 1.5

let size = vImageGetResamplingFilterSize(scale,
                                         kernelFunc,
                                         kernelWidth,
                                         vImage_Flags(kvImageNoFlags))

resamplingFilter = ResamplingFilter.allocate(byteCount: size,
                                             alignment: 1)

On return, resamplingFilter is a ResamplingFilter structure, allocated with the correct amount of uninitialized memory.

Create a Linear Resampling Filter

Call vImageNewResamplingFilterForFunctionUsingBuffer(_:_:_:_:_:_:) to create the resampling filter and populate resamplingFilter.

vImageNewResamplingFilterForFunctionUsingBuffer(resamplingFilter,
                                                scale,
                                                kernelFunc,
                                                kernelWidth,
                                                nil,
                                                vImage_Flags(kvImageNoFlags))

Scaling using a custom resampling filter is the same process as using the default Lanczos resampling:

var backColor = UInt8(0)

let height = Float(sourceBuffer.height)
let yTranslate = (height - height * scale) * 0.5
vImageVerticalShear_ARGB8888(&sourceBuffer,
                             &intermediateBuffer,
                             0, 0,
                             yTranslate,
                             0,
                             resamplingFilter,
                             &backColor,
                             vImage_Flags(kvImageNoFlags))

let width = Float(sourceBuffer.width)
let xTranslate = (width - width * scale) * 0.5
vImageHorizontalShear_ARGB8888(&intermediateBuffer,
                               &destinationBuffer,
                               0, 0,
                               xTranslate,
                               0,
                               resamplingFilter,
                               &backColor,
                               vImage_Flags(kvImageNoFlags))

The following shows the same image as used in the Lanczos example, also maginifed 30 times.

A scaled version of an image consisting of a field of small spots without ringing artifacts.

Using linear resampling avoids the ringing artifacts.

Free the Resampling Filter Memory

After you’re finished working with the resampling filter, it’s important that you free its allocated memory. Freeing the resampling filter memory is slightly different, depending on whether you’ve used the default or a custom filter. Use the following for the default:

vImageDestroyResamplingFilter(resamplingFilter)

For custom resampling filters, use the following:

resamplingFilter.deallocate()

See Also

vImage Operations

Adjusting the Brightness and Contrast of an Image

Use a gamma function to apply a linear or exponential curve.

Adjusting Saturation and Applying Tone Mapping

Convert an RGB image to discrete luminance and chrominance channels, and apply color and contrast treatments.

Blurring an Image

Filter an image by convolving it with custom and high-speed kernels.

Adding a Bokeh Effect

Simulate a bokeh effect by applying dilation.

Converting Color Images to Grayscale

Convert a color image to grayscale using matrix multiplication.

Standardizing Arbitrary Image Formats for Processing

Convert assets with disparate color spaces and bit depths to a standard working format for applying vImage operations.

Specifying Histograms with vImage

Calculate the histogram of one image and apply it to a second image.

Finding the Sharpest Image in a Sequence of Captured Images

Share image data between vDSP and vImage to compute the sharpest image from a bracketed photo sequence.

vImage Operations

Apply image manipulation operations to vImage buffers.