The Media layer contains the graphics, audio, and video technologies you use to implement multimedia experiences in your apps. The technologies in this layer make it easy for you to build apps that look and sound great.
High-quality graphics are an important part of all apps, and iOS provides numerous technologies to help put your custom art and graphics onscreen. The iOS graphics technologies offer a wide range of support, working seamlessly with the UIKit view architecture to make it easy to deliver content. You can use the standard views to deliver a high-quality interface quickly, or you can create your own custom views and use any of the technologies listed in Table 2-1 to deliver an even richer graphical experience.
UIKit defines high-level support for drawing images and Bézier paths and for animating the content of your views. In addition to providing classes to implement drawing support, UIKit views provide a fast and efficient way to render images and text-based content. Views can also be animated, both explicitly and using UIKit dynamics, to provide feedback and promote user interactivity.
For more information about the classes of the UIKit framework, see UIKit Framework Reference.
Core Graphics framework
Core Graphics (also known as Quartz) is the native drawing engine for iOS apps and provides support for custom 2D vector- and image-based rendering. Although not as fast as OpenGL ES rendering, this framework is well suited for situations where you want to render custom 2D shapes and images dynamically.
For more information, see Core Graphics Framework.
Core Animation (part of the Quartz Core framework) is a foundational technology that optimizes the animation experience of your apps. UIKit views use Core Animation to provide view-level animation support. You can use Core Animation directly when you want more control over the behavior of your animations.
For more information, see Quartz Core Framework.
Core Image provides advanced support for manipulating video and still images in a nondestructive manner.
For more information, see Core Image Framework.
OpenGL ES and GLKit
OpenGL ES handles advanced 2D and 3D rendering using hardware-accelerated interfaces. This framework is traditionally used by game developers, or by anyone wanting to implement an immersive graphical experience. This framework gives you full control over the rendering process and offers the frame rates needed to create smooth animations. For more information, see OpenGL ES Framework.
GLKit is a set of Objective-C classes that provide the power of OpenGL ES using an object-oriented interface. For more information, see GLKit Framework.
Metal provides extremely low-overhead access to the A7 GPU, enabling incredibly high performance for your sophisticated graphics rendering and computation tasks. Metal eliminates many performance bottlenecks—such as costly state validation—that are found in traditional graphics APIs. For more information, see Metal Framework.
TextKit and Core Text
TextKit is a family of UIKit classes used to perform fine typography and text management. If your app performs advanced text manipulations, TextKit provides seamless integration with the rest of your views. For more information, see Text Kit.
Core Text is a lower-level C-based framework for handling advanced typography and layout. For more information, see Core Text Framework.
Image I/O provides interfaces for reading and writing most image formats. For more information, see Image I/O Framework.
The Photos and PhotosUI frameworks provide access to a user’s photos, videos, and media. You use this framework in places where you want to integrate the user’s own content with your app. For more information, see Photos Framework and Photos UI Framework.
iOS provides built-in support for apps running on either Retina displays or standard-resolution displays. For vector-based drawing, the system frameworks automatically use the extra pixels of a Retina display to improve the crispness of your content. And if you use images in your app, UIKit provides support for loading high-resolution variants of your existing images automatically. For more information about what you need to do to support high-resolution screens, see App-Related Resources in App Programming Guide for iOS.
The iOS audio technologies work with the underlying hardware to provide a rich audio experience for your users. This experience includes the ability to play and record high-quality audio, to handle MIDI content, and to work with a device’s built-in sounds.
If your app uses audio, there are several technologies available for you to use. Table 2-2 lists these frameworks and describes the situations where you might use each.
Media Player framework
This high-level framework provides easy access to a user’s iTunes library and support for playing tracks and playlists. Use this framework when you want to integrate audio into your app quickly and when you don’t need to control playback behavior. For more information, see Media Player Framework.
AV Foundation is an Objective-C interface for managing the recording and playback of audio and video. Use this framework for recording audio and when you need fine-grained control over the audio playback process. For more information, see AV Foundation Framework.
OpenAL is an industry-standard technology for delivering positional audio. Game developers frequently use this technology to deliver high-quality audio using a set of cross-platform interfaces. For more information, see OpenAL Framework.
Core Audio is a set of frameworks that provide both simple and sophisticated interfaces for the recording and playback of audio and MIDI content. This framework is for advanced developers who need fine-grained control over their audio. For more information, see Core Audio.
iOS supports many industry-standard and Apple-specific audio formats, including the following:
Apple Lossless (ALAC)
DVI/Intel IMA ADPCM
Microsoft GSM 6.10
The iOS video technologies provide support for managing static video content in your app or playing back streaming content from the Internet. For devices with the appropriate recording hardware, you can also record video and incorporate it into your app. Table 2-3 lists the technologies that support video playback and recording.
The AVKit framework provides a set of simple-to-use interfaces for presenting video. This framework supports both full-screen and partial-screen video playback and supports optional playback controls for the user. For more information, see AVKit Framework.
AV Foundation provides advanced video playback and recording capabilities. Use this framework in situations where you need more control over the presentation or recording of video. For example, augmented reality apps could use this framework to layer live video content with other app-provided content. For more information, see AV Foundation Framework.
The Core Media framework defines the low-level data types and interfaces for manipulating media. Most apps do not need to use this framework directly, but it is available when you need unparalleled control over your app’s video content. For more information, see Core Media Framework.
iOS supports many industry-standard video formats and compression standards, including the following:
H.264 video, up to 1.5 Mbps, 640 by 480 pixels, 30 frames per second, Low-Complexity version of the H.264 Baseline Profile with AAC-LC audio up to 160 Kbps, 48 kHz, stereo audio in
H.264 video, up to 768 Kbps, 320 by 240 pixels, 30 frames per second, Baseline Profile up to Level 1.3 with AAC-LC audio up to 160 Kbps, 48 kHz, stereo audio in
MPEG-4 video, up to 2.5 Mbps, 640 by 480 pixels, 30 frames per second, Simple Profile with AAC-LC audio up to 160 Kbps, 48 kHz, stereo audio in
Numerous audio formats, including the ones listed in Audio Technologies
AirPlay lets your app stream audio and video content to Apple TV and stream audio content to third-party AirPlay speakers and receivers. AirPlay support is built into numerous frameworks—UIKit framework, Media Player framework, AV Foundation framework, and the Core Audio family of frameworks—so in most cases you do not need to do anything special to support it. Any content you play using these frameworks is automatically made eligible for AirPlay distribution. When the user chooses to play your content using AirPlay, it is routed automatically by the system.
Additional options for delivering content over AirPlay include the following:
To extend the content displayed by an iOS device, create a second window object and assign it to any
UIScreenobjects that are connected to the device through AirPlay. Use this technique when the content you display on the attached screen is different than the content displayed on the iOS device.
The playback classes of the Media Player framework automatically support AirPlay. You can also display Now Playing content on a connected Apple TV using AirPlay.
AVPlayerclass in AV Foundation to manage your app’s audio and video content. This class supports streaming its content via AirPlay when enabled by the user.
For web-based audio and video, you can allow that content to be played over AirPlay by including an
embedtag with the
UIWebViewclass also supports media playback using AirPlay.
For information on how to take advantage of AirPlay in your apps, see AirPlay Overview.
Media Layer Frameworks
The following sections describe the frameworks of the Media layer and the services they offer.
Assets Library Framework
The Assets Library framework (
AssetsLibrary.framework) provides access to the photos and videos managed by the Photos app on a user’s device. Use this framework to access items in the user’s saved photos album or in any albums imported onto the device. You can also save new photos and videos back to the user’s saved photos album.
For more information about the classes and methods of this framework, see Assets Library Framework Reference.
AV Foundation Framework
The AV Foundation framework (
AVFoundation.framework) provides a set of Objective-C classes for playing, recording, and managing audio and video content. Use this framework when you want to integrate media capabilities seamlessly into your app’s user interface. You can also use it for more advanced media handling. For example, you use this framework to play multiple sounds simultaneously and control numerous aspects of the playback and recording process.
The services offered by this framework include:
Audio session management, including support for declaring your app’s audio capabilities to the system
Management of your app’s media assets
Support for editing media content
The ability to capture audio and video
The ability to play back audio and video
Metadata management for media items
Precise synchronization between sounds
An Objective-C interface for determining details about sound files, such as the data format, sample rate, and number of channels
Support for streaming content over AirPlay
For more information about how to use AV Foundation, see AVFoundation Programming Guide. For information about the classes of the AV Foundation framework, see AV Foundation Framework Reference.
The AVKit framework (
AVKit.framework) leverages existing objects in AV Foundation to manage the presentation of video on a device. It is intended as a replacement for the Media Player framework when you need to display video content.
For more information about this framework, see the header files.
Core Audio is a family of frameworks (listed in Table 2-4) that provides native support for handling audio. These frameworks support the generation, recording, mixing, and playing of audio in your apps. You can also use these interfaces to work with MIDI content and to stream audio and MIDI content to other apps.
Defines the audio data types used throughout Core Audio. For more information, see Core Audio Framework Reference.
Provides playback and recording services for audio files and streams. This framework also provides support for managing audio files, playing system alert sounds, and triggering the vibrate capability on some devices. For more information, see Audio Toolbox Framework Reference.
Provides services for using the built-in audio units, which are audio processing modules. This framework also supports vending your app’s audio content as an audio component that is visible to other apps. For more information, see Audio Unit Framework Reference.
Provides a standard way to communicate with MIDI devices, including hardware keyboards and synthesizers. You use this framework to send and receive MIDI messages and to interact with MIDI peripherals connected to an iOS-based device using the dock connector or network. For more information, see Core MIDI Framework Reference.
Provides access to the audio tap interfaces.
For more information about Core Audio, see Core Audio Overview. For information about how to use the Audio Toolbox framework to play sounds, see Audio Queue Services Programming Guide.
The CoreAudioKit framework (
CoreAudioKit.framework) provides standard views for managing connections between apps that support inter-app audio. One view provides a switcher that displays the icons of other connected apps and the other view displays the transport controls that the user can use to manipulate the audio provided by the host app.
For more information about the interfaces of this framework, see the framework header files.
Core Graphics Framework
The Core Graphics framework (
CoreGraphics.framework) contains the interfaces for the Quartz 2D drawing API. Quartz is the same advanced, vector-based drawing engine that is used in OS X. It supports path-based drawing, antialiased rendering, gradients, images, colors, coordinate-space transformations, and PDF document creation, display, and parsing. Although the API is C based, it uses object-based abstractions to represent fundamental drawing objects, making it easy to store and reuse your graphics content.
For more information on how to use Quartz to draw content, see Quartz 2D Programming Guide and Core Graphics Framework Reference.
Core Image Framework
The Core Image framework (
CoreImage.framework) provides a powerful set of built-in filters for manipulating video and still images. You can use the built-in filters for everything from touching up and correcting photos to face, feature, and QR code detection. The advantage of these filters is that they operate in a nondestructive manner, leaving your original images unchanged. Because the filters are optimized for the underlying hardware, they are fast and efficient.
For information about the classes and filters of the Core Image framework, see Core Image Reference Collection.
Core Text Framework
The Core Text framework (
CoreText.framework) offers a simple, high-performance C-based interface for laying out text and handling fonts. This framework is for apps that do not use TextKit but that still want the kind of advanced text handling capabilities found in word processor apps. The framework provides a sophisticated text layout engine, including the ability to wrap text around other content. It also supports advanced text styling using multiple fonts and rendering attributes.
For more information about the Core Text interfaces, see Core Text Programming Guide and Core Text Reference Collection.
Core Video Framework
The Core Video framework (
CoreVideo.framework) provides buffer and buffer-pool support for the Core Media framework (described in Core Media Framework). Most apps never need to use this framework directly.
Game Controller Framework
The Game Controller framework (
GameController.framework) lets you discover and configure Made-for-iPhone/iPod/iPad (MFi) game controller hardware in your app. Game controllers can be devices connected physically to an iOS device or connected wirelessly over Bluetooth. The Game Controller framework notifies your app when controllers become available and lets you specify which controller inputs are relevant to your app.
For more information about supporting game controllers, Game Controller Programming Guide.
The GLKit framework (
GLKit.framework) contains a set of Objective-C based utility classes that simplify the effort required to create an OpenGL ES app. GLKit supports four key areas of app development:
GLKViewControllerclasses provide a standard implementation of an OpenGL ES–enabled view and associated rendering loop. The view manages the underlying framebuffer object on behalf of the app; your app just draws to it.
GLKTextureLoaderclass provides image conversion and loading routines to your app, allowing it to automatically load texture images into your context. It can load textures synchronously or asynchronously. When loading textures asynchronously, your app provides a completion handler block to be called when the texture is loaded into your context.
The GLKit framework provides implementations of vectors, matrices, and quaternions, as well as a matrix stack operation that provides the same functionality found in OpenGL ES 1.1.
GLKReflectionMapEffectclasses provide existing, configurable graphics shaders that implement commonly used graphics operations. In particular, the
GLKBaseEffectclass implements the lighting and material model found in the OpenGL ES 1.1 specification, simplifying the effort required to migrate an app from OpenGL ES 1.1 to later versions of OpenGL ES.
For information about the classes of the GLKit framework, see GLKit Framework Reference.
Image I/O Framework
The Image I/O framework (
ImageIO.framework) provides interfaces for importing and exporting image data and image metadata. This framework makes use of the Core Graphics data types and functions and supports all of the standard image types available in iOS. You can also use this framework to access Exif and IPTC metadata properties for images.
For information about the functions and data types of this framework, see Image I/O Reference Collection.
Media Accessibility Framework
The Media Accessibility framework (
MediaAccessibility.framework) manages the presentation of closed-caption content in your media files. This framework works in conjunction with new settings that let the user enable the display of closed captions.
For information about the contents of this framework, see the header files.
Media Player Framework
The Media Player framework (
MediaPlayer.framework) provides high-level support for playing audio and video content from your app. You can use this framework to do the following:
Play video to a user’s screen or to another device over AirPlay. You can play this video full screen or in a resizable view.
Access the user’s iTunes music library. You can play music tracks and playlists, search for songs, and present a media picker interface to the user.
Configure and manage movie playback.
Display Now Playing information in the lock screen and App Switcher. You can also display this information on an Apple TV when content is delivered via AirPlay.
Detect when video is being streamed over AirPlay.
For information about the classes of the Media Player framework, see Media Player Framework Reference. For information on how to use these classes to access the user’s iTunes library, see iPod Library Access Programming Guide.
Metal provides extremely low-overhead access to the A7 GPU enabling incredibly high performance for your sophisticated graphics rendering and computational tasks. Metal eliminates many performance bottlenecks—such as costly state validation—that are found in traditional graphics APIs. Metal is explicitly designed to move all expensive state translation and compilation operations out of the critical path of your most performance sensitive rendering code. Metal provides precompiled shaders, state objects, and explicit command scheduling to ensure your application achieves the highest possible performance and efficiency for your GPU graphics and compute tasks. This design philosophy extends to the tools used to build your app. When your app is built, Xcode compiles Metal shaders in the project into a default library, eliminating most of the runtime cost of preparing those shaders.
Graphics, compute, and blit commands are designed to be used together seamlessly and efficiently. Metal is specifically designed to exploit modern architectural considerations, such as multiprocessing and shared memory, to make it easy to parallelize the creation of GPU commands.
With Metal, you have a streamlined API, a unified graphics and compute shading language, and Xcode-based tools, so you don’t need to learn multiple frameworks, languages and tools to take full advantage of the GPU in your game or app.
For more information about how to use Metal, see Metal Programming Guide, Metal Framework Reference, and Metal Shading Language Guide.
The Open Audio Library (OpenAL) interface is a cross-platform standard for delivering positional audio in apps. You can use it to implement high-performance, high-quality audio in games and other programs that require positional audio output. Because OpenAL is a cross-platform standard, the code modules you write using OpenAL on iOS can be ported to many other platforms easily.
For information about OpenAL, including how to use it, see http://www.openal.org.
OpenGL ES Framework
The OpenGL ES framework (
OpenGLES.framework) provides tools for drawing 2D and 3D content. It is a C-based framework that works closely with the device hardware to provide fine-grained graphics control and high frame rates for full-screen immersive apps such as games. You use the OpenGL framework in conjunction with the EAGL interfaces, which provide the interface between your OpenGL ES drawing calls and the native window objects in UIKit.
The framework supports OpenGL ES 1.1, 2.0, and 3.0. The 2.0 specification added support for fragment and vertex shaders and the 3.0 specification added support for many more features, including multiple render targets and transform feedback.
For information on how to use OpenGL ES in your apps, see OpenGL ES Programming Guide for iOS. For reference information, see OpenGL ES Framework Reference.
The Photos framework (
Photos.framework) provides new APIs for working with photo and video assets, including iCloud Photos assets, that are managed by the Photos app. This framework is a more capable alternative to the Assets Library framework. Key features include a thread-safe architecture for fetching and caching thumbnails and full-sized assets, requesting changes to assets, observing changes made by other apps, and resumable editing of asset content.
For more information about the interfaces of this framework, see Photos Framework Reference.
Photos UI Framework
The Photos UI framework (
PhotosUI.framework) lets you create app extensions for editing image and video assets in the Photos app. For more information about how to create photo editing extensions, see App Extension Programming Guide.
Quartz Core Framework
The Quartz Core framework (
QuartzCore.framework) contains the Core Animation interfaces. Core Animation is an advanced compositing technology that makes it easy to create view-based animations that are fast and efficient. The compositing engine takes advantage of the underlying hardware to manipulate your view’s contents efficiently and in real time. Specify the start and end points of the animation, and let Core Animation do the rest. And because Core Animation is built in to the underlying
UIView architecture, it is always available.
For more information on how to use Core Animation in your apps, see Core Animation Programming Guide and Core Animation Reference Collection.
SceneKit is an Objective-C framework for building simple games and rich app user interfaces with 3D graphics, combining a high-performance rendering engine with a high-level, descriptive API. SceneKit has been available since OS X v10.8 and is now available in iOS for the first time. Lower-level APIs (such as OpenGL ES) require you to implement the rendering algorithms that display a scene in precise detail. By contrast, SceneKit lets you describe your scene in terms of its content—geometry, materials, lights, and cameras—then animate it by describing changes to those objects.
SceneKit’s 3D physics engine enlivens your app or game by simulating gravity, forces, rigid body collisions, and joints. Add high-level behaviors that make it easy to use wheeled vehicles such as cars in a scene, and add physics fields that apply radial gravity, electromagnetism, or turbulence to objects within an area of effect.
Use OpenGL ES to render additional content into a scene, or provide GLSL shaders that replace or augment SceneKit’s rendering. You can also add shader-based post-processing techniques to SceneKit’s rendering, such as color grading or screen space ambient occlusion.
For more information about the interfaces of this framework, see Scene Kit Framework Reference.
The SpriteKit framework (
SpriteKit.framework) provides a hardware-accelerated animation system for 2D and 2.5D games. SpriteKit provides the infrastructure that most games need, including a graphics rendering and animation system, sound playback support, and a physics simulation engine. Using SpriteKit frees you from creating these things yourself and lets you focus on the design of your content and the high-level interactions for that content.
Content in a SpriteKit app is organized into scenes. A scene can include textured objects, video, path-based shapes, Core Image filters, and other special effects. SpriteKit takes those objects and determines the most efficient way to render them onscreen. When it comes time to animate the content in your scenes, you can use SpriteKit to specify explicit actions you want to perform or use the physics simulation engine to define physical behaviors (such as gravity, attraction, or repulsion) for your objects.
In addition to the SpriteKit framework, there are Xcode tools for creating particle emitter effects and texture atlases. You can use the Xcode tools to manage app assets and update SpriteKit scenes quickly.
For more information about how to use SpriteKit, see SpriteKit Programming Guide. For an example of how to use SpriteKit to build a working app, see code:Explained Adventure.