The Major Parts of an Image Unit

An image processing filter, when packaged as an executable image unit, has three major parts:

• A kernel routine file. This file contains one or more kernel routines and any needed subroutines. The kernel routine must be written using the Core Image Kernel Language (CIKL). A C-like language, CIKL is a subset of the OpenGL Shading Language (glslang). CIKL restricts some of the glslang keywords that you can use, but introduces a number of keywords and data types that are not provided by glslang. (See Core Image Kernel Language Reference.)

• Objective-C filter files. Each filter has an interface and implementation file that performs all the set up work required prior to applying a kernel routine.

• Plug-in loading files. Each image unit has an interface and implementation file that implements the plug-in loading protocol.

Division of Labor

Image processing tasks are divided into low-level (kernel) and high-level (Objective-C) tasks. When you first start writing image units, you might find it challenging to divide the tasks appropriately. If you strive to keep kernel routines lean, you will likely succeed in dividing the tasks appropriately.

A kernel routine operates on individual pixels and uses the GPU (assuming one is available). For best performance, a kernel routine should be as focused as possible on pixel processing. Any set up work or calculations that can be done outside the kernel routine should be done outside the kernel routine, in the Objective-C filter files. As you’ll see, because Core Image expects certain tasks to be performed outside the kernel routine, the Xcode image unit plug-in template provides methods set up for just this purpose. In Writing the Objective-C Portion you see the specifics, but for now, a general understanding is all you’ll need.

These are the tasks typically performed in the Objective-C filter files:

• Retrieve the files needed for the filter. Image I/O is a high-level task that is typically done during the initialization phase of the image unit filter. Files can include the image (or images) to be processed and any other image data needed by the kernel routine (such as a texture or an environment map).

• Set up one or more CISampler objects. A sampler (lowercase “s”) is a data source for a kernel routine. (It is defined in Core Image Kernel Language Reference.) A CISampler object is a Core Image class that encapsulates a sampler, references a file to fetch samples from, defines a coordinate transform (if any) to use on the samples, and defines modes to use for interpolation and wrapping. The data source referenced by a CISampler object can be a texture, an environment map, an image to process, a lookup table—whatever is needed by the kernel routine.

• Set up one or more CIKernel objects. A kernel (lowercase “k”) refers to a kernel routine. (It is defined in Core Image Kernel Language Reference.) A CIKernel object is a Core Image class that encapsulates a kernel file, references each of the kernel routines in the file and defines a region-of-interest method for each of the kernel routines that requires such a method.

• Set a region-of-interest method and any input parameters required for that method. A region of interest (ROI) defines the area in the source image from which a sampler takes pixel information to provide to the kernel for processing. Simple filters—those for which there is a 1:1 mapping between a source and destination pixel—don’t need a method to calculate the region of interest because Core Image assumes a 1:1 mapping if you don’t supply an ROI method. See Region-of-Interest Methods for more details.

• Set up input parameters for the kernel routine. The Objective-C portion of an image unit is where you perform all calculations possible so that the values you pass to the kernel routine are ready to use. For example, you could calculate the radius of an effect in the Objective-C portion rather than pass a diameter to the kernel and perform the radius calculation in the kernel. This way, the calculation is performed only once, and not for every pixel that’s processed.

• Apply the kernel routine. You can invoke a kernel routine more than once (as you might for effects that require iteration, such as a blur effect). You can use more than one kernel routine to process an image. You can also combine your kernel routine with an effect produced by one of the built-in Core Image filters.

Kernel Routine Rules

A kernel routine is like a worker on an assembly line—it specializes in a focused task. Each time the routine is invoked, it produces a vec4 data type from the materials (input parameters) given to it. The routine must perform as little work as possible to be efficient. Assembly line work goes fastest when workers use preassembled subcomponents. It’s also true of kernel routines. Anything that can be calculated ahead of time and passed to the routine should be. As you become more experienced at writing kernel routines, you’ll devise clever ways to pare down the code in the routine. The examples in Writing Kernels should give you some ideas. Core Image also helps in this regard by restricting what sorts of operations can be done in a kernel routine.

Keep the following in mind as you design and write kernel routines:

• Flow control statements (if, for, while, do while) are supported only when the loop condition can be inferred at the time the code compiles.

• The input parameters to a kernel routine can be any of these data types: sampler, __table sampler, __color, float, vec2, vec3, or vec4. However, when you apply a kernel routine in the Objective-C portion of an image unit, you must supply objects. See Table 1-1.

• A kernel routine does not take images as input parameters. Instead, it takes a sampler object that’s associated with an image. It is the job of the sampler object to fetch image data and provide it to the kernel routine. All sampler objects are set up as CISampler objects in the Objective-C portion of an image unit. See Division of Labor.

• You are restricted to using what’s described in Core Image Kernel Language Reference. The Core Image Kernel Language (CIKL) is a subset of OpenGL Shading Language (glslang), so not everything that’s defined by glslang is allowed by CIKL. However, you’ll find that most of the keywords in gslang are available to you. In addition, CIKL provides a number of data types, keywords, and functions that are not available in glslang.

• You can’t use arrays.

• A kernel routine computes an output pixel by using an inverse mapping back to the corresponding pixels of the input images. Although you can express most pixel computations this way—some more naturally than others—there are some image processing operations for which this is difficult, if not impossible. For example, computing a histogram is difficult to describe as an inverse mapping to the source image. You also cannot perform seed fills or any image analysis operations that require complex conditional statements.

• A routine is faster if you unroll loops.

Table 1-1 lists the valid input parameters for a kernel routine and the associated objects that must be passed to the kernel routine from the Objective-C portion of an image unit. Core Image extracts the appropriate base data type from the higher-level Objective-C object that you supply. If you don’t use an object, the filter may unexpectedly quit. For example, if, in the Objective-C portion of the image unit, you pass a floating-point value directly instead of packaging it as an NSNumber object, your filter will not work. In fact, when you use the Image Unit Validator tool on such an image unit, the image unit fails with a cryptic message. (See Validating an Image Unit.)

Region-of-Interest Methods

The region of interest (ROI) is the area of the sampler source data that your kernel uses for its per-pixel processing. A kernel routine always returns a vec4 data type—that is, one pixel. However, the routine can operate on any number of pixels to produce that vec4 data type. If the mapping between the source and the destination is not one-to-one, then you must define a region-of-interest method in the Objective-C filter file.

You do not need an ROI method when a kernel routine:

• Processes a pixel from the working-space coordinate (r,s) of the sampler to produce a pixel at the working-space coordinate (r,s) in the destination image.

You must supply an ROI method when a kernel routine:

• Uses many source pixels in the calculation of one destination pixel. For example, a distortion filter might use a pixel (r,s) and its neighbors from the source image to produce a single pixel (r,s) in the destination image.

• Uses values provided by a sampler that are unrelated to the working-space coordinates in the source image and the destination. For example, a texture, a color ramp, or a table that approximates a function provides values that are unrelated to the notion of working coordinates.

You supply an ROI method for each kernel routine in an image unit that needs you. (An image unit can contain one or more kernel routines.) Each ROI method that you supply must use a method signature of the following form:

- (CGRect) regionOf:(int)samplerIndex

            destRect:(CGRect)r

            userInfo:obj;

You can replace regionOf with an appropriate name. For example, an ROI method for a blur filter could be named blurROI:destRect:userInfo:.

Core Image invokes your ROI method when appropriate, passing to it the sampler index (which you’ll learn more about later), the rectangle for the region being rendered, and any data that is needed by your routine. The method must define the ROI for each sampler data source used by the kernel routine. If a sampler data source used by the kernel routine doesn’t require an ROI method, then you can pass back the destRect rectangle for that sampler. You simply check the samplerIndex value passed to the method. If an ROI calculation is need for the sampler, perform the calculation and pass back the appropriate rectangle. If an ROI calculation is not needed for that particular sampler, then pass back the destRect passed to the method.

For example, if your kernel routine accesses pixels within a radius r around the current target, you need to offset the destination rectangle in the ROI method by the radius r. You’ll see how all this works in more detail later.

Next Steps

Now that you have a general idea of what the major parts of an image unit are and what each does, you are ready to move on to writing kernel routines. Writing Kernels starts with a few simple kernel routines and progresses to more complex ones. Not only will you se how to write kernel routines, but you’ll see how you can test simple kernel routines without the need to provide Objective-C code.