Sample Code

Converting Luminance and Chrominance Planes to an ARGB Image

Create a displayable ARGB image from the luminance and chrominance information supplied by your device’s camera.

Download

Overview

As an alternative to the technique discussed in Applying vImage Operations to Video Sample Buffers, vImage provides lower-level functions for creating RGB images from the separate luminance and chrominance planes provided by an AVCaptureSession instance. These functions offer better performance and more granular configuration than the vImageConvert_AnyToAny function.

This sample walks you through the steps for converting separate luminance and chrominance planes to an ARGB image:

  1. Configuring YpCbCr-to-ARGB Information.

  2. Defining the vImage destination buffer.

  3. Initializing the source buffers.

  4. Initializing the destination buffer.

  5. Converting the Yp and CbCr planes to ARGB.

  6. Displaying the result.

Configure YpCbCr-to-ARGB Information

The vImageConvert_YpCbCrToARGB_GenerateConversion(_:_:_:_:_:_:) function generates the information required to convert the luminance and chrominance planes to a single ARGB image. You supply it with a matrix containing the coefficients for the conversion and, in this example, kvImage_YpCbCrToARGBMatrix_ITU_R_601_4 contains the conversion matrix from ITU-R Recommendation BT.601-4.

The following code shows how to populate infoYpCbCrToARGB with the required conversion information for 8-bit pixels, clamped to a video range. An 8-bit video range format typically uses the range [16,235] for luminance and [16,240] for chrominance.

var infoYpCbCrToARGB = vImage_YpCbCrToARGB()

func configureYpCbCrToARGBInfo() -> vImage_Error {
    var pixelRange = vImage_YpCbCrPixelRange(Yp_bias: 16,
                                             CbCr_bias: 128,
                                             YpRangeMax: 235,
                                             CbCrRangeMax: 240,
                                             YpMax: 235,
                                             YpMin: 16,
                                             CbCrMax: 240,
                                             CbCrMin: 16)
    
    let error = vImageConvert_YpCbCrToARGB_GenerateConversion(
        kvImage_YpCbCrToARGBMatrix_ITU_R_601_4!,
        &pixelRange,
        &infoYpCbCrToARGB,
        kvImage422CbYpCrYp8,
        kvImageARGB8888,
        vImage_Flags(kvImageNoFlags))
    
    return error
}

Define the vImage Destination Buffer

To avoid reinitializing the destination RGB buffer for every video frame, declare it as a class member rather than declaring it inside the captureOutput(_:didOutput:from:) method of the sample buffer delegate:

var destinationBuffer = vImage_Buffer()

Initialize the Source Buffers from Pixel Buffer Planes

The source luminance and chrominance vImage buffers are initialized directly from the two planes of the pixel buffer. The first plane contains luminance information and the second plane contains chrominance information.

Use the following code to query pixel buffer properties for each plane and initialize your source buffers:

let lumaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
let lumaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
let lumaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)
let lumaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0)

var sourceLumaBuffer = vImage_Buffer(data: lumaBaseAddress,
                                     height: vImagePixelCount(lumaHeight),
                                     width: vImagePixelCount(lumaWidth),
                                     rowBytes: lumaRowBytes)

let chromaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)
let chromaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)
let chromaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)
let chromaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1)

var sourceChromaBuffer = vImage_Buffer(data: chromaBaseAddress,
                                       height: vImagePixelCount(chromaHeight),
                                       width: vImagePixelCount(chromaWidth),
                                       rowBytes: chromaRowBytes)

Initialize the Destination Buffer

Query the data property of the destination buffer you instantiated earlier, to find out if it needs to be initialized. The destination buffer will contain the RGB image after conversion. This code initializes destinationBuffer on the first pass and sets its size to match the luminance plane of the pixel buffer:

var error = kvImageNoError
if destinationBuffer.data == nil {
    error = vImageBuffer_Init(&destinationBuffer,
                              sourceLumaBuffer.height,
                              sourceLumaBuffer.width,
                              cgImageFormat.bitsPerPixel,
                              vImage_Flags(kvImageNoFlags))
    
    guard error == kvImageNoError else {
        return
    }
}

Convert Yp and CbCr Planes to ARGB

With the conversion information, destination buffers, and source buffers initialized, you’re ready to convert the two source buffers to the destination buffer by using the vImageConvert_420Yp8_CbCr8ToARGB8888(_:_:_:_:_:_:_:) function:

error = vImageConvert_420Yp8_CbCr8ToARGB8888(&sourceLumaBuffer,
                                             &sourceChromaBuffer,
                                             &destinationBuffer,
                                             &infoYpCbCrToARGB,
                                             nil,
                                             255,
                                             vImage_Flags(kvImagePrintDiagnosticsToConsole))

guard error == kvImageNoError else {
    return
}

Display the Result

To display the ARGB image to the user, create a Core Graphics image from destinationBuffer, and initialize a UIImage instance from that. The vImageCreateCGImageFromBuffer(_:_:_:_:_:_:) function returns an unmanaged CGImage instance based on the supplied buffer and the same format you used earlier.

Because captureOutput(_:didOutput:from:) runs in a background thread, you must dispatch the call to update the image view to the main thread.

let cgImage = vImageCreateCGImageFromBuffer(&destinationBuffer,
                                            &cgImageFormat,
                                            nil,
                                            nil,
                                            vImage_Flags(kvImageNoFlags),
                                            &error)

if let cgImage = cgImage, error == kvImageNoError {
    DispatchQueue.main.async {
        self.imageView.image = UIImage(cgImage: cgImage.takeRetainedValue())
    }
}

See Also

Conversion Between Image Formats

Building a Basic Conversion Workflow

Learn the fundamentals of the convert-any-to-any function by converting a CMYK image to an RGB image.

Converting Color Images to Grayscale

Convert a color image to grayscale using matrix multiplication.

Standardizing Arbitrary Image Formats for Processing

Convert assets with disparate color spaces and bit depths to a standard working format for applying vImage operations.

Conversion

Convert an image to a different format.