Share image data between vDSP and vImage to compute the sharpest image from a bracketed photo sequence.
- iOS 13.0+
- Xcode 11.2+
This sample code project captures a sequence of photographs and uses a combination of routines from vImage and vDSP to order the images by their relative sharpness. This technique is useful in applications such as an image scanner, where your user requires the least blurry captured image. After applying the routines, the app displays the images in a list, with the sharpest image at the top:
This sample walks you through the steps to find the sharpest image in a sequence of captured images:
Configure the capture session.
Define the photo settings for the sequence of captured images.
Acquire the images.
Initialize the grayscale vImage buffer.
Create floating point pixels to use in vDSP.
Convolve the image using a 3 x 3 single-pass edge detection Laplacian kernel (the result of this convolution pass is also shown in the app’s user interface).
Calculate the variance, or how spread out the pixel values are, in the convolved image.
Create a vImage buffer from the vDSP convolution result.
Create a display image with correct orientation.
Configure the Capture Session
The 3 x 3 Laplacian kernel used in this sample will report a lot of noise if applied to a full-resolution image. To reduce this noise, use a downscaled image by defining the capture session’s preset to a size that’s smaller than the camera’s native resolution:
To learn more about configuring a capture session, see Setting Up a Capture Session.
Define the Photo Settings
The sample defines the
AVCapture object, that specifies the capture features and settings, in
The sharpness detection algorithm in this sample works on a grayscale image. Use one of the camera’s YpCbCr pixel formats, either
k. These formats represent the luminance of the image using one plane, and color information on separate planes.
The following code checks that the current device supports one or both of these formats:
Create an array of
AVCapture instances and define the exposure target bias of each to
current. The maximum number of items in the array is defined by the
max property of the
Use the array of exposure settings and the first available YpCbCr format type to define the bracketed settings:
AVCapture instance to capture the sequence of images:
Acquire the Captured Image
For each captured image, AVFoundation calls the
pixel property of the
AVCapture instance supplied by AVFoundation to acquire the uncompressed CVPixelBuffer that contains the captured photograph. While your code is accessing the pixel data of the pixel buffer, use
CVPixel to lock the base address:
The pixel buffer vended by AVFoundation contains two planes; it is the plane at index zero that contains the luminance data. To run the sharpness detection code in a background thread, use
copy to create a copy of the luminance data:
You can now unlock the pixel buffer’s base address and pass the copied luminance data to the processing function in a background thread:
Initialize Grayscale vImage Source Buffer
Create a vImage buffer from data passed to the
source contains a grayscale representation of the captured image.
Create Floating Point Pixels to Use in vDSP
vImage buffers store their image data in row major format. However, when you are passing data between vImage and vDSP, be aware that, in some cases, vImage will add extra bytes at the end of each row. For example, the following code declares an 8-bit per pixel buffer that’s 10 pixels wide:
Although the code defines a buffer with 10 bytes per row, to maximize performance,
v will initialize a buffer with 16 bytes per row:
In some cases, this disparity between the row bytes used to hold image data and the buffer’s actual row bytes may not affect your app’s results. For this sample, compare the destination’s
row property against its
width, multiplied by the stride of a
Pixel and, if the values are the same, you can infer there’s no row byte padding and simply pass a pointer to the vImage buffer’s data to vDSP’s
However, in the case where there is row byte padding, create an intermediate vImage buffer with explicit row bytes and use the vImage
v function to populate
Perform the Convolution
The Laplacian kernel finds edges in the single-precision pixel values. Define the kernel as an array:
Use the vDSP convolve function to perform the convolution in-place on the
After the convolution, edges in the image have high values. The following image shows the result after convolution using the Laplacian kernel:
Calculate the Standard Deviation
v function to calculate the standard deviation of the pixel values after the edge detection:
std contains the standard deviation, and you use this value as a measure of relative sharpness. Images with more variance have more detail than those with less variance, and that difference is used to derive the relative sharpness.
Create a vImage Buffer from the vDSP Convolution Result
To display the result of the convolution, create a vImage buffer from the pixel data in
float. The following clips the result to
0 ... 255. The clipping ensures there’s no overflow when converting the single-precision values to unsigned 8-bit integers with
make. This function creates a new vImage buffer, using
preferred to compute the ideal row bytes for the image width:
make applies a gamma function to the Laplacian result to improve its visibility in the user interface. To learn more about using gamma functions in vImage, see Adjusting the Brightness and Contrast of an Image.
Create a Display Image with Correct Orientation
Use the vImage 90º rotation functions in conjunction with the
CGImage objects’s orientation to create a vImage buffer suitable for displaying in the app. The
static Blur function accepts a planar buffer (either the grayscale representation of the captured image or the result of the convolution) and the orientation, and returns a
For landscape images, that is images with an orientation of
.right, the function creates a destination buffer with a width equal to the height, and a height equal to the width of the supplied buffer. For portrait images, that is images with an orientation of
.down, the function creates a destination buffer with the same orientation as the supplied buffer:
The destination buffer is populated using
Finally, the function returns a
CGImage from the destination buffer: