Drawing With OpenGL ES

Framebuffer Objects Store Rendering Results

The OpenGL ES specification requires that each implementation provide a mechanism that an application can use to create a framebuffer to hold rendered images. On iOS, all framebuffers are implemented using framebuffer objects, which are built-in to OpenGL ES 2.0, and provided on all iOS implementations of OpenGL ES 1.1 by the GL_OES_framebuffer_object extension.

Framebuffer objects allow your application to precisely control the creation of color, depth, and stencil targets. You can also create multiple framebuffer objects on an single context, possibly sharing resources between the frame buffers.

To correctly create a framebuffer:

1. Create a framebuffer object.

2. Create one or more targets (renderbuffers or textures), allocate storage for them, and attach each to an attachment point on the framebuffer object.

3. Test the framebuffer for completeness.

Depending on what task your application intends to perform, your application configures different objects to attach to the framebuffer object. In most cases, the difference in configuring the framebuffer is in what object is attached to the framebuffer object’s color attachment point:

• If the framebuffer is used to perform offscreen image processing, attach a renderbuffer. See “Creating Offscreen Framebuffer Objects”

• If the framebuffer image is used as an input to a later rendering step, attach a texture. “Using Framebuffer Objects to Render to a Texture”

• If the framebuffer is intended to be displayed to the user, use a special Core Animation-aware renderbuffer. See “Rendering to a Core Animation Layer”

Rendering to a Core Animation Layer

Most applications that draw using OpenGL ES want to display the contents of the framebuffer to the user. On iOS, all images displayed on the screen are handled by Core Animation. Every view is backed by a corresponding Core Animation layer. OpenGL ES connects to Core Animation through a special Core Animation layer, a CAEAGLLayer. A CAEAGLLayer allows the contents of an OpenGL ES renderbuffer to also act as the contents of a Core Animation layer. This allows the renderbuffer contents to be transformed and composited with other layer content, including content rendered using UIKit or Quartz. Once Core Animation composites the final image, it is displayed on the device’s main screen or an attached external display.

In most cases, your application never directly allocates a CAEAGLLayer object. Instead, your application defines a subclass of UIView that allocates a CAEAGLLayer object as its backing layer. At runtime, your application instantiates the view and places it into the view hierarchy. When the view is instantiated, your application initializes an OpenGL ES context and creates a framebuffer object that connects to Core Animation.

The CAEAGLLayer provides this support to OpenGL ES by implementing the EAGLDrawable protocol. An object that implements the EAGLDrawable works closely with an EAGLContext object. A drawable object provides two key pieces of functionality. First, it allocates shared storage for a renderbuffer. Second, it works closely with the context to present that renderbuffer’s content. Presenting the contents of a renderbuffer is loosely defined by EAGL; for a CAEAGLLayer object, presenting means that the renderbuffer’s contents replace any previous contents presented to Core Animation. An advantage of this model is that the contents of the Core Animation layer do not need to be rendered every frame, only when the rendered image would actually change.

It is illustrative to walk through the steps used to create an OpenGL ES-aware view. The OpenGL ES template provided by Xcode implements this code for you.

1. Subclass UIView to create an OpenGL ES view for your iOS application.

2. Override the layerClass method so that your view creates a CAEAGLLayer object as its underlying layer. To do this, your layerClass method returns the CAEAGLLayer class.

   + (Class) layerClass

   {

   return [CAEAGLLayer class];

   }

3. In your view’s initialization routine, read the layer property of your view. Your code uses this when it creates the framebuffer object.

   myEAGLLayer = (CAEAGLLayer*)self.layer;

4. Configure the layer’s properties.

   For optimal performance, mark the layer as opaque by setting the opaque property provided by the CALayer class to YES. See “Improving Compositing Performance in Core Animation.”

   Optionally, configure the surface properties of the rendering surface by assigning a new dictionary of values to the drawableProperties property of the CAEAGLLayer object. EAGL allows you to specify the pixel format for the renderbuffer and whether it retains its contents after they are presented to the Core Animation. For a list of the keys you can set, see EAGLDrawable Protocol Reference.

5. Allocate a context and make it the current context. See “Configuring OpenGL ES Contexts.”

6. Create the framebuffer object (see above).

7. Create a color renderbuffer. Allocate its storage by calling the context’s renderbufferStorage:fromDrawable: method, passing the layer object as the parameter. The width, height and pixel format are taken from the layer and used to allocate storage for the renderbuffer.

   GLuint colorRenderbuffer;

   glGenRenderbuffers(1, &colorRenderbuffer);

   glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

   [myContext renderbufferStorage:GL_RENDERBUFFER fromDrawable:myEAGLLayer];

   glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorRenderbuffer);

8. Retrieve the height and width of the color renderbuffer.

   GLint width;

   GLint height;

   glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &width);

   glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &height);

   In earlier examples, the width and height of the renderbuffers was explicitly provided to allocate storage for the buffer. Here, the code retrieves the width and height from the color renderbuffer after its storage is allocated. Your application does this because the actual dimensions of the color renderbuffer are calculated based on the view’s bounds and scale factor. Other renderbuffers attached to the framebuffer must have the same dimensions. In addition to using the height and width to allocate the depth buffer, use them to assign the OpenGL ES viewport as well as to help determine the level of detail required in your application’s textures and models. See “Supporting High-Resolution Displays Using OpenGL ES.”

9. Allocate and attach a depth buffer.

10. Test the framebuffer object.

To summarize, the steps to create the framebuffer are almost identical. Each allocates a color attachment and a depth attachment, differing primarily in how the color attachment is allocated.

Table 4-1  Mechanisms for allocating the color attachment of the framebuffer

Offscreen renderbuffer	glRenderbufferStorage
Drawable renderbuffer	renderbufferStorage:fromDrawable:
Texture	glFramebufferTexture2D

Drawing to a Framebuffer Object

Now that you have a framebuffer object, you need to fill it. This section describes the steps required to render new frames and present them to the user. Rendering to a texture or offscreen framebuffer acts similarly, differing only in how your application uses the final frame.

Generally, applications render new frames in one of two situations:

• On demand; it renders a new frame when it recognizes that the data used to render the frame changed.

• In an animation loop; it assumes that data used to render the frame changes for every frame.

displayLink = [myView.window.screen displayLinkWithTarget:self selector:@selector(drawFrame)];

[displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];

Rendering a Frame

Figure 4-2 shows the steps an OpenGL ES application should take on iOS to render and present a frame. These steps include many hints to improve performance in your application.

Erase the Renderbuffers

At the start of every frame, erase all renderbuffers whose contents from a previous frames are not needed to draw the next frame. Call the glClear function, passing in a bit mask with all of the buffers to clear, as shown in Listing 4-2.

Listing 4-2  Erasing the renderbuffers

glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);

glClear(GL_DEPTH_BUFFER_BIT | GL_COLOR_BUFFER_BIT);

Not only is using glClear more efficient than erasing the buffers manually, but using glClear hints to OpenGL ES that the existing contents can be discarded. On some graphics hardware, this avoids costly memory operations to load the previous contents into memory.

Prepare OpenGL ES Objects

This step and the next step is the heart of your application, where you decide what you want to display to the user. In this step, you prepare all of the OpenGL ES objects — vertex buffer objects, textures and so on — that are needed to render the frame.

Execute Drawing Commands

This step takes the objects you prepared in the previous step and submits drawing commands to use them. Designing this portion of your rendering code to run efficiently is covered in detail in “OpenGL ES Application Design Guidelines.” For now, the most important performance optimization to note is that your application runs faster if it only modifies OpenGL ES objects at the start of rendering a new frame. Although your application can alternate between modifying objects and submitting drawing commands (as shown by the dotted line), it runs faster if it only performs each step once.

Resolve Multisampling

If your application uses multisampling to improve image quality, your application must resolve the pixels before they are presented to the user. Multisampling applications are covered in detail in “Using Multisampling to Improve Image Quality.”

Discard Unneeded Renderbuffers

A discard operation is defined by the EXT_discard_framebuffer extension and is available on iOS 4.0 and later. Discard operations should be omitted when your application is running on earlier versions of iOS, but included whenever they are available. A discard is a performance hint to OpenGL ES; it tells OpenGL ES that the contents of one or more renderbuffers are not used by your application after the discard command completes. By hinting to OpenGL ES that your application does not need the contents of a renderbuffer, the data in the buffers can be discarded or expensive tasks to keep the contents of those buffers updated can be avoided.

At this stage in the rendering loop, your application has submitted all of its drawing commands for the frame. While your application needs the color renderbuffer to display to the screen, it probably does not need the depth buffer’s contents. Listing 4-3 discards the contents of the depth buffer.

Listing 4-3  Discarding the depth framebuffer

const GLenum discards[] = {GL_DEPTH_ATTACHMENT};

glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);

glDiscardFramebufferEXT(GL_FRAMEBUFFER,1,discards);

Present the Results to Core Animation

At this step, the color renderbuffer holds the completed frame, so all you need to do is present it to the user. Listing 4-4 binds the renderbuffer to the context and presents it. This causes the completed frame to be handed to Core Animation.

Listing 4-4  Presenting the finished frame

glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

[context presentRenderbuffer:GL_RENDERBUFFER];

By default, you must assume that the contents of the renderbuffer are discarded after your application presents the renderbuffer. This means that every time your application presents a frame, it must completely recreate the frame’s contents when it renders a new frame. The code above always erases the color buffer for this reason.

If your application wants to preserve the contents of the color renderbuffer between frames, add the kEAGLDrawablePropertyRetainedBacking key to the dictionary stored in the drawableProperties property of the CAEAGLLayer object, and remove the GL_COLOR_BUFFER_BIT constant from the earlier glClear function call. Retained backing may require iOS to allocate additional memory to preserve the buffer’s contents, which may reduce your application’s performance.

Supporting View Controllers in Your OpenGL ES Application

Many OpenGL ES applications are immersive; they take over the entire screen to display their content. However, those same applications often need to interact with other features of iOS. For example, applications that want to display iAd advertisements or use Game Center’s built-in view controllers must provide a view controller. This view controller is used to modally display the contents provided by those system view controllers. For this reason, the Xcode template now provides a corresponding view controller to manage the view. Your application should do the same.

An important use for a view controller is to handle view rotations. On PowerVR SGX-equipped devices running iOS 4.2 and later, the performance of Core Animation rotation and scaling of OpenGL ES content has been significantly improved. Your application should use a view controller to set the allowed orientations and to transition between orientations when the user rotates the device, as described in View Controller Programming Guide for iOS. By using a view controller, other view controllers presented modally by your view controller are presented in the same orientation as your view controller.

If your application supports PowerVR MBX-equipped devices, on those devices you may need to avoid Core Animation; instead, perform the rotations directly inside OpenGL ES. When a rotation occurs, change the modelview and projection matrices and swap the width and height arguments to the glViewport and glScissor functions.

Improving Compositing Performance in Core Animation

The contents of renderbuffers are animated and composited along with any other Core Animation layers in your view hierarchy, regardless of whether those layers were drawn with OpenGL ES, Quartz or other graphics libraries. That’s helpful, because it means that OpenGL ES is a first-class citizen to Core Animation. However, mixing OpenGL ES content with other content takes time; when used improperly, your application may perform too slowly to reach interactive frame rates. The performance penalties for mixing and matching content vary depending on the underlying graphics hardware on the iOS device; devices that use the PowerVR MBX graphics processor incur more severe penalties when compositing complex scenes. For best results, alway test your application on all iOS devices you intend it to ship on.

For the absolute best performance, your application should rely solely on OpenGL ES to render your content. To do this, size the view that holds your CAEAGLLayer object to match the screen, set its opaque property to YES, and ensure that no other Core Animation layers or views are visible. If your OpenGL ES layer is composited on top of other layers, making your CAEAGLLayer object opaque reduces but doesn’t eliminate the performance cost.

If your CAEAGLLayer object is blended on top of layers underneath it in the layer hierarchy, the renderbuffer’s color data must be in a premultiplied alpha format to be composited correctly by Core Animation. Blending OpenGL ES content on top of other content has a severe performance penalty.

Using Multisampling to Improve Image Quality

Starting in iOS 4, OpenGL ES supports the APPLE_framebuffer_multisample extension. Multisampling is a form of antialiasing, allowing your application to smooth jagged edges and improve image quality. Multisampling uses more resources (memory and fragment processing), but improves image quality in most 3D applications.

Figure 4-3 shows how multisampling works in concept. Instead of creating one framebuffer object, your application now creates two. The first framebuffer object is the resolve buffer; it contains a color renderbuffer, but otherwise is allocated exactly as you did before. The resolve buffer is where the final image is rendered. The second framebuffer object, the multisampling buffer, contains both depth and color attachments. The multisample renderbuffers are allocated using the same dimensions as the resolve framebuffer, but each includes an additional parameter that specifies the number of samples to store for each pixel. Your application performs all of its rendering to the multisampling buffer and then generates the final antialiased image by resolving those samples into the resolve buffer.

Listing 4-5 shows the code to create the multisampling buffer. This code uses the width and height of the previously created buffer. It calls the glRenderbufferStorageMultisampleAPPLE function to create multisampled storage for the renderbuffer.

Listing 4-5  Creating the multisample buffer

glGenFramebuffers(1, &sampleFramebuffer);

glBindFramebuffer(GL_FRAMEBUFFER, sampleFramebuffer);

glGenRenderbuffers(1, &sampleColorRenderbuffer);

glBindRenderbuffer(GL_RENDERBUFFER, sampleColorRenderbuffer);

glRenderbufferStorageMultisampleAPPLE(GL_RENDERBUFFER, 4, GL_RGBA8_OES, width, height);

glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, sampleColorRenderbuffer);

glGenRenderbuffers(1, &sampleDepthRenderbuffer);

glBindRenderbuffer(GL_RENDERBUFFER, sampleDepthRenderbuffer);

glRenderbufferStorageMultisampleAPPLE(GL_RENDERBUFFER, 4, GL_DEPTH_COMPONENT16, width, height);

glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, sampleDepthRenderbuffer);

if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE)

    NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));

Here are the steps to modify your rendering code to support multisampling:

1. During the Erase Buffers step, you clear the multisampling framebuffer’s contents.

   glBindFramebuffer(GL_FRAMEBUFFER, sampleFramebuffer);

   glViewport(0, 0, framebufferWidth, framebufferHeight);

   glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

2. After submitting your drawing commands, you resolve the contents from the multisampling buffer into the resolve buffer. The samples stored for each pixel are combined into a single sample in the resolve buffer.

   glBindFramebuffer(GL_DRAW_FRAMEBUFFER_APPLE, resolveFrameBuffer);

   glBindFramebuffer(GL_READ_FRAMEBUFFER_APPLE, sampleFramebuffer);

   glResolveMultisampleFramebufferAPPLE();

3. In the Discard step, you can discard both renderbuffers attached to the multisample framebuffer. This is because the contents you plan to present are stored in the resolve framebuffer.

   const GLenum discards[] = {GL_COLOR_ATTACHMENT0,GL_DEPTH_ATTACHMENT};

   glDiscardFramebufferEXT(GL_READ_FRAMEBUFFER_APPLE,2,discards);

4. In the Presentation step presents the color renderbuffer attached to the resolve framebuffer.

   glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

   [context presentRenderbuffer:GL_RENDERBUFFER];

Multisampling is not free; additional memory is required to store the additional samples, and resolving the samples into the resolve framebuffer takes time. If you add multisampling to your application, always test your application’s performance to ensure that it remains acceptable.

Supporting High-Resolution Displays Using OpenGL ES

If your application uses OpenGL ES for rendering, your existing drawing code should continue to work without any changes. When drawn on a high-resolution screen, though, your content is scaled accordingly and will appear as it does on non-Retina displays. The reason for this is that the default behavior of the CAEAGLLayer class is to set the scale factor to 1.0. To enable high-resolution drawing, you must change the scale factor of the view you use to present your OpenGL ES content. For more information on how high-resolution displays are supported in UIKit, see “Supporting High-Resolution Screens In Views”.

Changing the contentScaleFactor property of your view triggers a matching change to the scale factor of the underlying CAEAGLLayer object. The renderbufferStorage:fromDrawable: method, which you use to create the storage for the renderbuffer, calculates the size of the renderbuffer by multiplying the layer’s bounds by its scale factor. The code provided in “Rendering to a Core Animation Layer” allocates the renderbuffer’s storage in Core Animation, then retrieves the width and height of renderbuffer; this allows it to automatically pick up any changes to the scale of its content. If you double the scale factor, the resulting renderbuffer has four times as many pixels to render. After that, it is up to you to provide the content to fill those additional pixels.

If you opt in to high-resolution drawing, adjust the model and texture assets of your application accordingly. When running on an iPad or a high-resolution device, you might want to choose more detailed models and textures to render a better image. Conversely, on a standard-resolution iPhone, you can continue to use smaller models and textures.

An important factor when determining whether to support high-resolution content is performance. The quadrupling of pixels that occurs when you change the scale factor of your layer from 1.0 to 2.0 causes four times as many fragments to be processed. If your application performs many per-fragment calculations, the increase in pixels may reduce your application’s frame rate. If you find your application runs significantly slower at a higher scale factor, consider one of the following options:

• Optimize your fragment shader’s performance using the performance-tuning guidelines found in this document.

• Implement a simpler algorithm in your fragment shader. By doing so, you are reducing the quality of individual pixels to render the overall image at a higher resolution.

• Use a fractional scale factor between 1.0 and and the screen’s scale factor. A scale factor of 1.5 provides better quality than a scale factor of 1.0 but needs to fill fewer pixels than an image scaled to 2.0.

• Use multisampling instead. An added advantage is that multisampling also provides higher quality on devices that do not support high-resolution displays.

• Create your color and depth renderbuffers using the lowest precision types that still provide good results; this reduces the memory bandwidth required to operate on the renderbuffers.

The best solution depends on the needs of your OpenGL ES application; during development, test these options and choose the approach that provides the best balance between performance and image quality.

Creating OpenGL ES Contexts on an External Display

Some iOS devices can be attached to an external display. The resolution of an external display and its content scale factor may differ from the main screen; your code that renders a frame should adjust to match.

The procedure for creating your context is almost identical to that running on the main screen. Follow these steps:

1. Follow the steps found in “Displaying Content on an External Display” in View Programming Guide for iOS to create a new window on the external display.

2. Add your OpenGL ES view to the window.

3. Create a display link object by retrieving the screen property of the window and calling its displayLinkWithTarget:selector:. This creates a display link optimized for that display.