Welcome to session 506.
My name is Tim Monroe, and I am a member
of the AVFoundation engineering Team, and I want to talk
to you today about editing movies in AVFoundation.
The takeaway message is very simple.
AVFoundation provides new classes
for editing QuickTime movie files.
Oh, come on.
You can do better than that.
Let's hear some applause.
Now, I am excited about this capability because,
as you'll see in a little bit, this provides the tools
that I need to manage a personal project that I have been working
on for about the last ten years, and in some sense,
it brings to a culmination a project that I have been doing
for about the last 35 years.
So I've got a lot of skin in this game,
and I hope you'll enjoy the presentation today.
So what does it mean to say we can edit QuickTime movie files.
Well, we can open QuickTime movie files
and perform range-based editing on movies and tracks.
So you select a section, a segment of a movie,
and copy it into some other movie.
We can add and remove tracks.
We can set tracks associations between tracks --
so say to mark one track at a chapter track of another.
We can add or modify track and movie metadata.
And we can create movie files and what I am going
to call URL sample reference movie files.
So how are we going to do this today?
First thing I am going to do is situate these new classes
in respect to the existing classes we have in AVFoundation.
Then I am going to do sort of an API crawl where I go through
and touch on the high points
of the new methods we've added to these classes.
And finally, I am going to go back and talk
about this personal project that I've mentioned
and show you very clearly how these new capabilities allow me
to do interesting things with this large amount
of data that I've collected.
So let's talk about the AVFoundation editing classes.
If you've done any work with AVFoundation
in the editing space, you know that we have two classes
that you work with, the operative abstraction is
that of a composition, and so you can have AVCompositions
and at the mutable level, AVMutableCompositions.
Now, compositions are really cool stuff.
You can make some really exciting presentations.
There's just one pesky little drawback
which is there's no standard file format
into which you can serialize your composition.
So it makes it hard
to interchange your snazzy composition
with other applications.
Well, when we move to AVMovie and AVMutableMovie,
that problem goes away because this is tied
to the QuickTime movie file format.
We can now open these, edit them, and write them back.
At the track level, we have a similar setup.
As you know, a composition is composed of composition tracks,
and at the mutable level,
we have AVMutableComposition tracks.
In El Capitan, we are adding two new classes at the track level,
AVMovieTrack and AVMutableMovieTrack.
So we have, really, five new classes that you have
to worry about, movie and mutable movie, movie track
and mutable movie track,
and then this fifth class called AVMediaDataStorage.
That's a very simple class, and its whole job in life is
to indicate where you want new sample data that you are writing
into a file to end up.
So an AVMovie represents the data in an audiovisual file
that conforms to the QuickTime movie file format.
Now, if you are familiar with QuickTime movies,
you know that there are also a number of related file formats
that were devised based on the QuickTime movie file format,
and these are collectively called ISO base media
And AVMovie and mutable movie can handle these just as well
as they can QuickTime movie files.
Now, what's the key feature about these files?
From my point of view, it's this,
that the file format enforces a strict separation
between sample data and the data that organizes that sample data
into tracks and movies.
So here's a basic diagram of a QuickTime movie file.
It starts with what's called a file type box,
and this is a very simple little bit of data that indicates
which of the file formats
in this family this file conforms to.
There's also a movie box which, as I just said,
is the organizing information about all that sample data,
which is found in its own box.
Now, the order of most of this is arbitrary.
The file type box has to come first,
but the other boxes can come in pretty much any order.
So we could have the movie box at the end of the file,
it could be situated between a couple of sample data boxes,
there can even be parts of the file that are unused.
Maybe they used to contain data, but they don't anymore.
Now, what's in this movie box?
Well, there are three types of information that we are going
to be concerned with here today.
First is what I like to call global settings.
This is information that without
which you wouldn't really have a movie,
things like how many tracks are in the movie,
what's the duration of the movie,
when was this file created, and what's the preferred rate
that you would like this played back at.
There's also in a movie box some metadata, which is sort
of optional data that isn't strictly speaking necessary
for the playback of the media, but is useful to have.
And this would be stuff like a copyright statement,
an indication of who the author, what the title is,
and maybe even some custom metadata that you
or some other app has written into this movie box.
The last thing that you will find
in a movie box are track boxes, and these are pieces
of information that define the tracks in the movie.
So they contain the track type,
they point to the sample data that's needed for the track.
They also contain track metadata.
And again, you can write custom track metadata
in there if you'd like.
And there's also information about track association,
so how this track relates to other tracks in the movie.
So the key feature here is that the sample data location --
so the sample data itself is not contained in the track box.
The track box contains references
to the sample data, sort of like this.
Now, it's possible for the sample data that a track refers
to exist in another file altogether,
so you can have external sample references.
It's even possible for the sample references to refer only
to external sample data.
Now, when we have this situation, the little movie box
and its file type box is called a sample reference movie file.
So there's no actual data in that box;
it just points to data elsewhere.
Now, it turns out that this is an incredibly powerful tool
to have available when you are working with media files.
Why is that?
Because I can edit that small little bit of information
and never have to touch that huge amount of media data.
However, as you can imagine,
there's a certain fragility involved here.
Why? Because I'm referring to data, and if the data
that I am referring to happens to move
or gets deleted, I am out of luck.
I can't play my presentation anymore.
Now, it's possible to reduce
that fragility in a couple of ways.
Primarily by using relative URLs
when I make these sample references, and we'll see how
to do that in a little bit.
But at the end of the day, you are going to want
to have what's called a self-contained movie.
If you want to give it to a friend to view or post it
for delivery on the Web.
And when you need to do that,
you can just run your mutable movie
through an AVAssetExportSession
to get a nice self-contained file.
Okay. So let's look at the movie editing API
that we are introducing.
At the immutable level -- so AVMovie is an immutable subclass
of AVAsset -- we can actually do some interesting things already.
We can inspect the movie.
We can get what's called the movie header.
And we can write this movie header into a new file.
By movie header, I just mean the file type box
and the movie box together, none of the sample data.
So how do we initialize an AVMovie?
If you have initialized an AVRL, you know how to do it.
You give us a URL, and we give you back an AVMovie.
You can also, if you have a movie header, say,
on the paste board, you can access that
and create a movie from a block of data.
We are not actually going to talk about this anymore
in this talk, but if you look
at our sample code package called AVMovieEditor,
you will see very nicely how you can put stuff
and take stuff off of the paste board.
So here is how easy it is
to create a sample reference movie file.
We'll open a URL as an AVMovie.
Sorry. Then we call the right --
an indication of the type of file we want to create,
and then some options.
Now, in this case, the option
that we've specified is truncate destination
to movie header only.
Essentially, this means if there's any data already
at that output URL, it's going to get blown away
and what's going to end up in
that file is simply the movie header.
Now, it's possible to pass in a different option
and not blow away any data that might be in that file,
so that would be the add movie header to destination option.
So let's move to the mutable subclass
of AVMovie, AVMutableMovie.
This provides editing methods that aus
to perform range-based movie editing, add and remove tracks,
add or modify movie metadata.
So how do we initialize an AVMutableMovie?
Pretty much the same way we initialize an AVMovie.
We will pass it a URL, but in this case we need
to be able toened had ale any errors that might occur.
It's also possible to create an AVMutableMovie from scratch.
So this line of code gives us back an empty AVMutableMovie
that we are going to want to add tracks to and then some data
to the tracks and so forth.
These are the segment editing methods available
If you have worked with AVMutable composition,
this is familiar because they are, in fact,
identical with one exception,
which is the insert time range method takes an additional
parameter that indicates whether you want to copy the sample data
into the target or just copy sample references.
Now, if you copy sample data into the target mutable movie,
you need to tell it where to write that sample data.
It does nothing by default.
It's not going to write it into the file
that the movie box is located in, for instance.
So you have to explicitly specify
where you want any new sample data that's written
into this movie to be put.
And you do that by specifying the default media data storage
property of the mutable movie.
And this is where this AVMediaDataStorage class comes
Again, it just simply wraps a URL at this point.
There are methods for creating and removing tracks,
and you see to create a track,
we have to say what type of track we want.
Do we want a video track,
do we want an audio track, and so forth.
And if we like, if we have an existing track that we want
to model this new one on, we can pass it in as an --
as a parameter and say take the relevant properties
from that existing track and use those.
So here's a simple little case study.
To update an existing movie file, we could open it
up from a URL, do some edits using the methods we've just
seen, and then we can use the right movie header to URL method
to write the movie header back into that same file.
This is what's known as in-place editing.
So here we haven't moved any sample data.
All we've done is tweak the movie box
and put it back into the same file.
So let's talk about the track editing API.
This allows us to modify tracks that are
in a QuickTime movie file,
and we have range-based track editing,
just like we saw at the movie level.
We can also set associations between tracks, and we can add
or modify track metadata.
So here are the segment editing methods,
and they should look familiar because we've just seen them.
At the movie level.
The only difference being that instead of insert time range
of asset, we have insert time range of track.
And again, when we do insert time range, we have to specify,
do you want to copy the sample data, or do you just want
to put in sample references.
And if you copy the sample data, the track has to know
where that sample data is going to go,
and we do that by setting the media data storage property
of that track using exactly the same class we used
at the movie level.
So here is another case study.
Suppose there's a little bit of a clip
that I want to silence out.
Maybe there's some unfortunate language in that clip.
I could do it it quite simply by finding the first audio track
in the movie; define a time range, which is the range I want
to silence; remove that time range;
and then insert an empty time range at that same spot.
The net effect of this code is to have an empty segment
where there used to be some offensive language, say.
Working with track associations is very simple.
You can add a track association or remove a track association.
We will see this in use later.
And I said I would tell you how to use relative URLs
when you copy sample data into your track,
and you do it like this.
You open. For each track,
we will set sample reference base URL property
to some common parent of the movie that contains the movie --
file that contains the movie box and the files
that contain all the media data.
So let's move on to this personal project
that I talk about, and I want to call it A Study
in Scarlet (and Gray) apologies to Conan Doyle.
Let's jump back in time to 1980.
What are we talking?
35 years ago.
At that time, I was a graduate student across the bay
at Berkeley, and I needed a way to blow off some energy,
so I went out and bought a pair of skates, and I discovered
that Berkeley is a great place to be a skater
because it's got these really nice hills.
And I really enjoyed that.
So I kept finding higher and higher hills to climb up
and then go flying down.
So here in 1984, you can see me at the top
of Claremont avenue in Oakland.
In the distance, you can see San Francisco, just this side
of the bay, we have Emeryville, and we have this wonderful,
almost looks like a ski run in front of me.
I am not quite sure what the hand up is doing.
Maybe it's the sort of we who are
about to die salute you thing, and I didn't die.
[Laughter] Well, what I did in order to keep track
of where I was, I went out and I bought a street map,
and I got myself a yellow highlighter,
and I started marking off on my map exactly where I had been,
which hills were particularly good.
Well, if you are like me, that's a really bad thing
to start doing because you're going to want to end
up coloring in the entire map.
And so about 2000 -- sorry.
About 1985, I came up with a project
that I called Tim's Radical Inline Skate Tour of Oakland,
or TRISTO, and the goal was really simple,
to skate every street in Oakland,
the entire length of it.
This continued from 1985 until about 2005,
so ten years ago, and I made it.
Oakland has about 800 miles of roadways that you would have
to skate in order to accomplish this.
Now, when I was done in 2005, I had no location data.
GPS hadn't been invented when I started,
and it certainly wasn't available.
Actually, I did have location data because if you open
up my map, you will see that it is right side up all yellow,
not just -- but Alameda, Piedmont,
Emeryville, the whole nine yards.
I also had no video.
If you remember back to that time, I mean,
the camera you would be carrying in 1985 would be clunky
and fragile, and you just wouldn't want to have it
with you when you are out there skating.
So when I finished this, I really --
this is a wacko project, I admit that,
and I made absolutely no effort to tell anybody about this.
Unfortunately, the San Francisco chronicle learned of this feat,
I guess, and wrote a nice little newspaper article,
one of these human interest stories.
The Oakland City Council learned about it,
they passed a resolution praising the extraordinary
physical accomplishment -- that's their words, not mine.
A couple of filmmakers in Oakland heard about it
and spent literally years off
and on making a nice little documentary about this project,
which came out about a year ago and has been making the rounds
of independent theaters in the Bay area.
Just completely wacko.
So about a couple of years after I finished this project,
I started spending some time on the east coast, in particular
in the area of Boston, Massachusetts,
and even more particular, in Cambridge.
And I thought, all right, let's take this thing on the road.
I spent a few years skating Cambridge and Somerville,
and then finally the idea for TRISTO Boston was born.
Skate every street in the city of Boston.
So this began about 2011.
I looked at a map, and I figured it would take me
about five years to complete this.
Well, we had a couple mild winters in there.
It's about the same size as Oakland.
And I was able to complete this in about two and a half years,
so in May 2013, I finished TRISTO Boston.
Now, this time I was prepared because as I was skating,
I had one of those sports action cameras up on the top
of my helmet, and I had a GPS app for my iPhone.
And so when I ended up,
I had 490 MPEG-4 files straight off the camera covering
about 200 different skates for a total
of about a terabyte and a half of data.
I also had GPS data in the form of GPX files --
that's sort of an XML version of location data --
we will see one later --
for about 150 megabytes worth of data.
So just to get an idea of what these movies look like,
let's just look at one.
So these are movies straight off the camera.
The only thing I have done is rename them
so that it's got the date
and title making it easy for me to find.
So we will open these up in QuickTime Player.
There's the second one, so that continues at the end
of the first one, so we will select the first movie here.
We can see it's a bright, sunny day.
It's late November.
It's a weekday.
And let's just roll this.
We see it's the fall because there's leaves on the ground.
If we had the audio, we would hear
that it's actually quite windy on this day.
And there's something strange here, though.
There's nobody on the street.
There's no pedestrians.
Let's jump ahead a little bit again.
There's some cars.
Here I am at Longwood Medical District.
This is a heavily trafficked area.
Here's a commercial district.
Nobody. It's like a ghost town.
So what's going on?
Well, we'll solve that problem later.
So what I've got, basically, is a hay stack
and a bunch of needles.
I've got 500 hours of video.
Even if 99% of it is junk, that's still five full hours
of interesting video that I'd like to find.
Part of the problem, though,
is I don't even know what the needles look
like because I don't know what's important
in these 500 hours of video.
Well, I do know some things that I'd really love to look at,
and here is sort of a montage of some of these.
I mean, come on.
Like, I didn't see that pothole?
Now, that one's my favorite, because as you can see,
I go down, I kind of roll over, I get back and keep on skating.
You know, if it wasn't as -- thank you.
Now, I actually had to pay somebody to weed --
to look through about 1/50 of all of my video
to get these six clips.
So in theory, there's 300
of these wonderful falls somewhere in that hay stack.
So at about, you know, five seconds per fall,
we are talking 25 minutes of this stuff,
which would be really fun to watch.
So here's what I want to do with all this data.
I've got a terabyte and a half of video data that I want
to somehow do something with.
I want to find those needles.
So the first step is going to be to combine the various files
that came off the camera for one skate
into a sample reference movie file so that I can just open it
and edit it and not have to worry about the camera files.
The second thing I want to do is add indexing metadata
as movie metadata.
So I want to go in and mark in the file, okay,
this is where I start to fall, and this is the end of the fall.
Or this is where dogs are chasing me,
and this is where they finally figured
out they are not going to get me.
Or this is where I am talking to a law enforcement officer,
and that just keeps going.
So I'd like to index all this stuff so that it's easy
to search and find all these needles, as I call them.
I want to add GPS data.
I have all that location data I haven't done anything with.
I want to add it to the file, and I am going to do
that as what's called a timed metadata track.
And the key is I want to do all of this without modifying any
of the original camera files,
and by minimizing any data copying that I need to do.
Okay. How are we going to do that?
Well, AVMovie and AVMutableMovie give us the answer.
The solution to the first issue is very simple.
I am going to take those camera files, and I am going
to create a sample reference movie file that points
to the data in those two original files.
Very straightforward, standard use
of sample reference movie files.
The solution for the second step is just to add a little bit
of extra custom metadata to my movie box.
The third step is to add some actual media data
into my new sample reference movie file.
And that media data, even though it's called timed metadata,
it's stored as media data or sample data, and it just goes
into a new track in the file.
So that's the way I want to approach this issue.
So the first step is actually quite simple.
We'll open the first movie as a mutable movie,
and we'll open the second
and any subsequent movies as just URL assets.
We'll define the range in the asset that we want to insert,
and that's just the entire asset duration.
We will actually insert it into the mutable movie at the end
of the mutable movie, and notice we say don't copy the
I just want to end up with sample references.
Finally, all we have to do is write a new movie header
out to a new file, and we have then created a sample reference
movie file that points to the data
in the two original camera files.
So let's look at some demos
of how I solve the second and third one.
Normally this slide would just read demos, but I am a sucker
for a good palindrome, and I just couldn't waste it.
Alvin is a sort of acronym
for AVFoundation-based linear indexer.
Its job is to do the indexing I talked about,
but also get the location data into my file.
This is what the UI looks like.
I've got a movie view on the left, I've got a map view
on the right, and down below you can see I've got some display
areas where I am going to be displaying information
that I glean about this skate.
So let's go ahead and open up a movie file.
And this is the sample reference movie file that I just created.
That opens up there.
I am going to load a GPX file.
I will go out and select the appropriate GPS file.
Now, when I open this file,
some of the data's going to be filled in.
It's going to figure out how long this was.
Okay? So let's -- I've got it so I can expand my map view
to make it clear where we are.
So I went out and made a big circuit around Brookline.
This situates you.
There is Logan Airport.
Here is Boston proper.
Now let's just zoom back in.
Oh, I said I was going to show you a GPX file.
So this is what a GPX file looks like.
It's just XML formatted, longitudes, latitudes,
elevations, time stamps.
That's it, nothing more in there.
So let's go here and run a Web service.
I am going to go out and find
out from a government Web server what the weather was
like on this day, and I discover it was quite windy,
it was 17 mile an hour average, or for those who speak metric,
28.4, and it was 39 degrees Fahrenheit or,
interesting factoid, 3.9 centigrade.
Thanksgiving day, that's why there's nobody on the street.
You see, it figured out this this was a holiday.
It's also the second night of Hanukkah.
Okay? So just these Web Services have allowed me
to deduce why there's nobody on the street.
Now I am running another Web Service called reverse
geocoding, where I am sending each longitude and latitude
out to a server at Apple, and it's going
to hand me back a state, city, street,
and sometimes even a street number.
Basically, it's going to give me information
about the streets that I was on.
So we will wait here while all these street pins fall down.
Then we can select one of them to find out what street that is,
so that's Commonwealth Avenue.
Or here's Washington Street in Brookline.
Now, I added a search bar so I could type in, say,
Commonwealth Avenue, and it will highlight it when it finds it.
So if I put this data into my file,
now I have complete searchability on street names.
I can go in and find all those little clips where I am
on a particular street.
So let's hide those, and the last thing we need
to do is synchronize our video with our location data,
and I tried various automatic ways to do this,
but I discovered the only really good way to do it was to go
to a frame in the video where I knew where I was on the map,
and then I am going to come over onto the map and click it
and say synchronize the video to this location right here.
Now, when I do this, if I come back
and start playing the video,
you will see the pin moves along with the video.
See, these are now synchronized.
And I can even go the other direction.
I can go to someplace on the map, and I can click it
and say make the movie go there, and now I am
on Commonwealth Avenue,
and I know this is a particularly fun downhill slope,
and we will just let this ride for a second.
And this is really why we do this, right, for these kind
of downhill with no cars and what could possibly go wrong?
Okay. So that's Alvin.
You can see that Alvin has solved the second
and third problems for me.
It's added custom metadata to my sample reference movie file.
And it basically does it like this.
It gets the existing metadata by calling the --
looking at the metadata property.
We'll create a new metadata item,
we'll fill in its properties accordingly.
For instance, the average wind speed value is the value I got
from the government server telling me what the weather was
on that day.
And I just create some identifier here that I can look
for later, and once I've written this into the file
with the right movie header to URL, my movie is all updated.
Now let's look at step three,
which is to add the location data as a time metadata track.
You already know how to do this.
If you were here last year
and heard the talk Harnessing Metadata in Audiovisual Media,
they showed a couple things.
First how to get location data from a capture session,
and how to right-time metadata into a movie.
Well, we've got our location data.
It's in our GPX file.
So we don't need that capture session stuff.
We can just focus on taking the source code
in that second sample code package
and putting it into Alvin.
Now, once we've done that, what we end up with
on disk is a timed file that has a single metadata track in it,
and all we have to do is copy that track into a new track
in our movie, and we will do it by opening the GPS asset.
We'll then add a new track
to our movie using whatever media type happens to be
in that track, and we will copy the settings from that track.
Finally, we insert that range into this new track, and again,
if we write this header into our file, we are done.
Now, the last thing we need to do -- this is optional --
is create a track association between the timed media track
and whatever track it is metadata for, and in this case,
we have chosen the video track.
If you leave this step out, the timed video track is assumed
to apply to the file as a whole, and we are fine
as far as that's concerned.
If you are working with AVMutableMovie,
there are a couple of things to keep in mind.
As you know, an AVMovie or an AVMutableMovie is an AVAsset
because it's a subclass.
So anything that you can do
with an AVAsset you can do with a mutable movie.
For instance, you could play it using AVPlayerItem,
you could grab an image from it using an AVAssetImageGenerator.
You could export it using an AVAssetExportSession.
Now, if you do any of these operations on a mutable movie,
we highly recommend that you make a copy
of that mutable movie because if you're changing the movie while
you are exporting it, bad things will happen.
So just make a copy of it, do your export,
and everything works fine.
The second best practice is that if you are opening an asset
that you want to insert into an AVMutableMovie,
you should set this flag, AV URL asset prefer precise duration
and timing key, to true.
Now, we haven't done this in any of the code snippets
that you've seen so far because when you open a QuickTime movie,
this flag is already set to true.
But if you were to open an MP3 file, for instance,
and wanted to insert that into a mutable movie,
you would want to set this flag.
So let's summarize.
We've added new classes to support editing
on QuickTime movie files.
This gives us, as I hope you've seen,
a really nice, simplified workflow.
We are able to work on reference movies and not
on the original movies themselves.
So if you are handling large amounts
of data, this is for you.
If you want to see how this works in practice,
download AVMovieEditor and take a look at that.
For more information, go to the standard places.
And if you have questions about this, we'll be in the lab
on Thursday and Friday.
If you want to look at the --
how to work with compositions, jump back to 2010.
And finally, if you want to see in great detail how
to do the timed metadata track generation,
go to last year's session,
Harnessing Metadata in Audiovisual media.
Thank you very much.
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.