Article

Optimizing Image Processing Performance

Improve your app's performance by converting image buffer formats from interleaved to planar.

Overview

vImage operates on two image buffer formats:

Interleaved. Stores each pixel's color data consecutively in a single buffer. For example, the data that describes a 4-channel image (red, green, blue, and alpha) would be stored as RGBARGBARGBA…

Planar. Stores each color channel in separate buffers. For example, a 4-channel image would be stored as four individual buffers containing red, green, blue, and alpha data.

Diagram showing how the color information for each pixel in an image is stored in interleaved and planar buffers.

Because many vImage functions operate on a single color channel at a time—by converting an interleaved buffer to planar buffers—you can often improve your app's performance by doing this conversion manually. However, most vImage functions are available in both the interleaved and planar variants, so before you do the conversion, try both to see which works better in your context.

In some cases, you may not want to apply a vImage operation to all four channels of an image. For example, you may know beforehand that the alpha channel is irrelevant in the images that you’re dealing with, or perhaps all of your images are grayscale and you only need to operate on one channel. Using planar formats makes it possible to isolate and work with only the channels you need.

Review Interleaved Performance

Typically, your source imagery is in interleaved format and your default option will be to use the interleaved variant of a vImage function. For example, the following code populates an interleaved 8-bit ARGB destination buffer with the result of a table lookup transformation on an 8-bit ARGB source buffer:

var lookUpTable = (0...255).map {
    return Pixel_8(($0 / 75) * 75)
}

vImageTableLookUp_ARGB8888(&sourceBuffer,
                           &destinationBuffer,
                           nil,
                           &lookUpTable,
                           &lookUpTable,
                           &lookUpTable,
                           vImage_Flags(kvImageNoFlags))

For information on logging CPU performance, see Logging.

Convert an Interleaved Source Buffer to Planar Buffers

First create a planar buffer for each color component of the source image you want to work with. The following code interrogates format—the vImage_CGImageFormat structure that represents the source image format—to calculate the number of components. (To learn more about Core Graphics image formats, see Creating a Core Graphics Image Format.)

let componentCount = format.componentCount

You can now create an array of initialized buffers for each color component. Note that each planar buffer's pixelBits value is set to the source image's bitsPerComponent value.

var argbSourcePlanarBuffers: [vImage_Buffer] = (0 ..< componentCount).map { _ in
    guard let buffer = try? vImage_Buffer(width: Int(sourceBuffer.width),
                                          height: Int(sourceBuffer.height),
                                          bitsPerPixel: format.bitsPerComponent) else {
                                            fatalError("Error creating source buffers.")
    }
    
    return buffer
}

Assuming your source image contains four 8-bit channels, the vImageConvert_ARGB8888toPlanar8(_:_:_:_:_:_:) function populates the planar buffers with the contents of the interleaved source image:

vImageConvert_ARGB8888toPlanar8(&sourceBuffer,
                                &argbSourcePlanarBuffers[0],
                                &argbSourcePlanarBuffers[1],
                                &argbSourcePlanarBuffers[2],
                                &argbSourcePlanarBuffers[3],
                                vImage_Flags(kvImageNoFlags))

Initialize the Destination Planar Buffers

Creating the destination buffers is similar to creating the source buffers:

var argbDestinationPlanarBuffers: [vImage_Buffer] = (0 ..< componentCount).map { _ in
    guard let buffer = try? vImage_Buffer(width: Int(sourceBuffer.width),
                                          height: Int(sourceBuffer.height),
                                          bitsPerPixel: format.bitsPerComponent) else {
                                            fatalError("Error creating destination buffers.")
    }
    
    return buffer
}

Derive the Alpha Channel Index

Because you don't want to apply the lookup table transform to the alpha channel, you need to derive the index of the alpha channel from the source image's alphaInfo property. The following code populates the alphaIndex variable with the index of the alpha channel:

let alphaIndex: Int?

let littleEndian = cgImage.byteOrderInfo == .order16Little ||
                   cgImage.byteOrderInfo == .order32Little

switch cgImage.alphaInfo {
    case .first, .noneSkipFirst, .premultipliedFirst:
        alphaIndex = littleEndian ? componentCount - 1 : 0
    case .last, .noneSkipLast, .premultipliedLast:
        alphaIndex = littleEndian ? 0 : componentCount - 1
    default:
        alphaIndex = nil
}

Copy the Source Alpha to the Destination Alpha

Use the alphaIndex value to copy the alpha information directly from the appropriate source buffer to the appropriate destination buffer:

if let alphaIndex = alphaIndex {
    do {
        try argbSourcePlanarBuffers[alphaIndex].copy(destinationBuffer: &argbDestinationPlanarBuffers[alphaIndex])
    } catch {
        fatalError("Error copying alpha buffer: \(error.localizedDescription).")
    }
}

Apply the Lookup Table to the Planar Buffers

To apply the lookup table transform to the color planar buffers, execute vImageTableLookUp_Planar8(_:_:_:_:) on each one:

for index in 0 ..< componentCount where index != alphaIndex {
    vImageTableLookUp_Planar8(&argbSourcePlanarBuffers[index],
                              &argbDestinationPlanarBuffers[index],
                              &lookUpTable,
                              vImage_Flags(kvImageNoFlags))
}

Convert the Planar Buffers Back to an Interleaved Buffer

With the planar buffers populated with the transform operation results, you're ready to convert the buffers back to an interleaved image. Create an interleaved destination buffer that matches the source buffer:

guard var destinationBuffer = try? vImage_Buffer(width: Int(sourceBuffer.width),
                                                 height: Int(sourceBuffer.height),
                                                 bitsPerPixel: format.bitsPerPixel) else {
                                                    fatalError("Error creating destination buffers.")
}

The vImageConvert_Planar8toARGB8888(_:_:_:_:_:_:) function interleaves the four planar buffers, writing the results to the destination buffer:

vImageConvert_Planar8toARGB8888(&argbDestinationPlanarBuffers[0],
                                &argbDestinationPlanarBuffers[1],
                                &argbDestinationPlanarBuffers[2],
                                &argbDestinationPlanarBuffers[3],
                                &destinationBuffer,
                                vImage_Flags(kvImageNoFlags))

Free the Buffer Memory

After you’re finished working with the buffers, it’s important that you free the memory allocated to them:

destinationBuffer.free()
for buffer in argbSourcePlanarBuffers {
    buffer.free()
}
for buffer in argbDestinationPlanarBuffers {
    buffer.free()
}