Debugging AVFoundation Compositions, Video Compositions, and Audio Mixes
This document discusses some common problems you may encounter when creating complex AVFoundation compositions, video compositions and audio mixes, and how to debug them.
The AVFoundation framework provides a feature-rich set of classes to facilitate the editing of audiovisual assets. At the heart of AVFoundation’s editing API are compositions, defined by the
AVComposition object. A composition is simply a collection of tracks from one or more different media assets. The
AVMutableComposition class provides an interface for inserting and removing tracks, as well as managing their temporal orderings. The tracks in an
AVComposition object are fixed.
If you want to perform any custom audio or video processing on the tracks in your composition, you need to incorporate an audio mix or a video composition, respectively. An
AVAudioMix object manages the input parameters for mixing audio tracks. It allows custom audio processing to be performed on audio tracks during playback or other operations. You can also set the relative volumes and ramping of audio tracks.
A video composition describes, for any time in the aggregate time range of its instructions, the number and IDs of video tracks that are to be used in order to produce a composed video frame corresponding to that time. When AV Foundation’s built-in video compositor is used, the instructions a video composition contains can specify a spatial transformation, an opacity value, and a cropping rectangle for each video source, and these can vary over time via simple linear ramping functions. An
AVVideoComposition object represents an immutable video composition. The AV Foundation framework also provides a mutable subclass,
AVMutableVideoComposition, that you can use to create new videos.
It’s quite possible to construct elaborate and complex compositions, video compositions, and audio mixes with entirely valid values that produce unexpected results. It's possible, for example, for a valid video composition to produce frames from which the contents of the source video tracks are entirely absent.
This document discusses some common pitfalls when constructing these, and ways to more easily debug them.
Here are some of the common types of problems you may encounter when building complex compositions, video compositions and audio mixes.
Misaligned track segments
The alignment of the tracks might be wrong. For example, this can happen as a result of CMTime value rounding errors when inserting tracks into a composition for a specific time.
Video Compositions, Audio Mixes
Gaps between segments
Don’t leave gaps between the segments in a video composition. This will almost certainly result in black frames, or a continuation of the last frame.
Misaligned layer instructions
There could be a misalignment between the video composition time ranges and the composition track segments.
Misaligned opacity/audio ramps
You might have made a mistake in one of the ramps — by making it too long, for example.
Bogus layer transforms
You may have an error in the layer transformation matrix. This can cause your output to spin off beyond the boundaries of the output frame.
Static frame for the entire duration of the source asset
You may have a video composition that produces a single static frame for the entire duration of the source asset. This can occur if you set the
frameDuration of the
AVMutableVideoComposition to the asset's duration. In the processing by the video compositor, only one frame would be produced, at the start time, and then "held" for the specified duration, in spite of the instructions for varying opacities, transforms, etc.
Debugging Compositions, Video Compositions and Audio Mixes
What’s the best way to debug a complex composition, video composition and audio mix?
Visualize the Composition
One technique that works well is to visualize these. Rather than looking at your code, you look at a picture of your composition, video composition and audio mix.
There’s sample code that has been designed just for this purpose:
These sample applications implement a custom
AVCompositionDebugView class which presents a visual description of the underlying
AVAudioMix objects which form the composition. See Figure 7.
Simply drop the
AVCompositionDebugView into your own app. It’s a non-interactive view. Extend it to draw your own video instructions. It will help you spot any overlaps and gaps in the composition tracks, video instructions, and the audio mix.
Debugging Video Compositions
AVVideoCompositionValidationHandling protocol declares methods that you can implement in the delegate of an
AVVideoComposition object to indicate whether validation of a video composition should continue after specific errors have been found:
In the course of validation with the isValid(for:timeRange:validationDelegate:) method, the receiver will invoke its
validationDelegate with reference to any trouble spots in the video composition.
You should implement the methods defined by the
AVVideoCompositionValidationHandling protocol to aid in the debugging of a video composition.
For more information about the editing of audiovisual assets with AVFoundation, see the Editing chapter of the AV Foundation Programming Guide.
Document Revision History
New document that discusses techniques for debugging complex AVFoundation Compositions, Video Compositions, and Audio Mixes