Delivering Intuitive Media Playback with AVKit

Welcome to Delivering Intuitive Media Playback with AVKit. My name is Jed Lewison, and I'll be joined a little bit later by my colleague Dan Wright. And we're going to talk with you about media playback with AVKit. And what's new in best practices with AVPlayerViewController on iOS and tvOS.

AVKit is a cross platform media playback UI framework built on top of AVFoundation in CoreMedia. And our mission is to make it easy for you to show and play AVPlayer-based media content using the same user interface used by Apple's own apps.

For UIKit apps, we provide AVPlayerViewController.

For APKit apps, we provide AVPlayerView. And for both UIKit and APKit apps, we provide AVRoutePickerView, which lets you add a wireless route picker to your custom playback UI's.

So let's take a look at some code. Getting started with AVPlayerViewController on iOS is incredibly easy, and tvOS. The first thing you do is you create an AVPlayer.

Then, you create and AVPlayerViewController and assign to it the player that you just created.

And finally, you present it. And that's all there is to it.

That's all it takes to get a rich and fully functional media playback UI that will be familiar to your users because it's the same UI used by Apple in its own apps. But there's a lot of power below the surface. And that's because when you use AVKit, you own the media playback objects. You are in control of creating and managing them. And video rendering is built on top of the same core technology that powers AVPlayerLayer.

That means you get the full power of AVFoundation in CoreMedia playback UI's, including all the power delivered by AVPlayer, AVPlayerItem, and AVAsset. But because AVKit also sit on top of UIKit and AppKit, you get a user experience tailored to each of Apple's unique platforms. So AVKit gives you the best of both worlds.

You get full control over your media playback objects. And a familiar API for showing and displaying them using UIKit and APKit-based APIs.

So, that's a high-level look at AVKIt. Let's dive right into what's new in AVPlayerViewController and iOS for iOS 13. I'm going to start talking about new full screen callbacks that we added that inform you about transitions from embedded inline presentations to full screen and entering full screen and exiting full screens. So, to motivate this, in this video, what we'll see is the user's going to enter full screen here and then start an interactive dismissal. But, they'll cancel that dismissal. So now we're back to full screen. And now the user has dismissed, and we're back to where we started with an AVPlayerViewController embedded inline.

So, the new API we're delivering in iOS 13 informs you about all of these states by extending AVPlayerViewController Delegate.

You get notifications when a full screen presentation begins or ends. And it looks like this.

Two new delegate methods.

If you link against the iOS 13 SDK, they will be available starting with iOS 12, one for beginning a full screen presentation, and one that is called when a full screen presentation ends.

So, let's take a look at an implementation here. And the key object is the UIViewControllerTransition Coordinator. At that works exactly the same as it does with any other view controller transition in UIKit. And you can learn about all those details and the power there from UIKit documentation. But the key thing I want to focus on here is the completion handler for the animate alongsideTransition callback. And this is where you find out whether a transition succeeded or whether the user cancelled it. And this is the source of truth for learning about entering full screen from an inline presentation.

And obviously it's interesting to know whether the player view controller is full screen or whether it is embedded inline. But why is it crucial? It's important because, let's take the case of where your player view controller may have been in a scroll view. And the user has entered full screen. When the device rotates, as you know, sometimes your scroll view's offset might change. And it's possible that the player view controller will scroll offscreen.

And that's okay for it to scroll offscreen during the full screen presentation because the user can't see it. But you need to make sure that the player view controller does stay alive. If it is deallocated, the full screen presentation will end. So you can use the new API to know when to take a strong reference to the player view controller if your table view controller or collection view happens to remove the player view controller if it's scrolled offscreen. But you're still left with the question of how do you get it back in place when the user exits full screen. And for that, you use the willEnd variant of the new API. And this is called the before the animation begins. So you can reset the location of the player view controller in your UI. And the user will be none the wiser. Everything will be back the way they expect.

So those are full screen callbacks.

Now let's take a look at AVPlayerViewController in iPad apps on the Mac. And as you know, iPad apps can now be built on the Mac. And AVPlayerViewController is a full participant in that.

And here's what it looks like. You get the same user interface that you expect from Apple's playback UIs with all the functionality that AVPlayerViewController delivers for iPad apps. But you also get some platform specific features for Mac. Things like touch bar support, which you get for free.

And keyboard support.

And we've also added picture-in-picture support to iPad apps on the Mac. In fact, we've extended picture-in-picture support to all UIKit and APPKit-based apps on the Mac for this release.

And how many new lines of code to you need to use AVPlayerViewController in an iPad app on the Mac? You guessed it. None.

So that's AVPlayerViewController and iPad apps on the Mac.

Now let's talk about external metadata. And to motivate this, imagine that your AirPlaying content. And imagine when you did that, your locked screen looked like this, instead of like this. And there's two things that make it possible to show this UI. The first thing is you need to handle remote control commands. And AVKit handles that for you automatically for free. You don't have to do anything.

But you also sometimes have media that doesn't have all the metadata that you would want to publish baked in. And for that, we're adding this new API to iOS. It's actually existed on tvOS. And you take your AVPlayerItem. And you can supplement metadata with things like title, metadata, or artwork metadata. It's super easy. And when you do that, we'll take care of the rest. We've also, in iOS 13, improved support for interactive -- for custom controls from AVPlayerViewController. That's always been possible to do, use custom controls with AVPlayerViewController on iOS. But when you did that, you lost all the other things that weren't onscreen. Things like interactive dismissals. Or landscape support.

Things like keyboard. And now in AVPlayerViewController on iPad apps for the Mac, touch bar support. So, what you can do now in iOS 13 is if you set showsPlaybackControls to False.

And then you present AVPlayerViewController modally.

And finally use the contentOverlayView as a place where you place your custom controls.

You can have a completely custom playback UI but still take advantage of things like nowPlayingInfo support and interactive dismissals. But there's some things, if you do this, some things you should keep in mind. You still are going to need to take care of status bar and home indicator appearance. And you should always pass unhandled touches through your view hierarchy so that AVKit can handle things like interactive dismissals or double-tap for zoom, which we'll talk about in a little bit.

So, that's a look at what's new in AVPlayerViewController on iOS. There's a bunch more things that we've added.

And one of them is we've made great strides in improving performance in iOS 12. And it's even better on iOS 13. So if you're interested in using AVPlayerViewController in scroll view, it should be buttery smooth now.

So, let's talk about some best practices with AVPlayerViewController.

I want to talk about three different ways that we show video on our devices. And of course, there's a fourth way that we show video from an iOS device AirPlay. And there was a terrific talk on Tuesday morning at 9:00 a.m. If you haven't seen it, I would strongly recommend watching that talk. But I'm going to focus on on-device playback today.

And let's start with full screen video playback. And of course, what I mean by that is full screen in the UIKit sense of the word. So, we're talking about covering your UIWindowScene coordinate space. And if we think about it, there's two different reasons why we might play a video full screen. It could be for sort of a background video in a splash screen scenario. Or because the video is the star of the show. And we want full playback controls. And let's take a quick look at the splash screen scenario. And you're going to have set of these things that you're going to want to implement in your application.

And to do that, you're going to take the playerViewController. You're going to embed it as a child. So it can be at the bottom of your view hierarchy. You're going to do things like disabling playback controls and setting the video gravity so that the video scales to fill the full screen.

If you're using the new video with alpha APIs that are new in this release, you're going to want to set your view's background color to clear or some other color to leverage that.

And there's some things to take into account with AVFoundation. You always want to make sure to prevent external playback with something that is playing as a splash screen. Because you don't want to interrupt, say, music playing on someone's Apple TV. And you want to configure the audio session. And especially this last property on AVAudioSession, that this secondary audio should be silenced. It lets you know if the user is playing audio from some other application. And they want that application to be -- that audio to be primary. In that case, if you have audio in your splash screen video, make sure to mute it.

But then, let's look at full screen playback where the video is the star of the show. And when we do this, of course we want playback UI that adapts beautifully whether it's a portrait presentation like this where we have a stacked playback transport controls. Or we're in landscape and we need a slightly different layout but the same -- but we need to deliver the same functionality. So AVPlayerViewController delivers this to you for free. And it also does things like, you'll notice in those two screen shots the video was unzoomed in portrait. And now it's zoomed in landscape.

And it's zoomed to fill the entire screen because the user has generally, in this case, has played video that is of similar aspect ratio to the device aspect ratio in a zoomed format. But some users prefer to have video unzoomed, like here. And when that happens we want to make sure that the video isn't occluded by any physical characteristic of the device. AVKit is aware of all these things and takes care of that for you when you're using PlayerViewController. But rather than running through every feature that we provide, I just want to zoom in on two very small things that we added in iOS 13. The first -- in this video, you're going to hear audio playing. And the user's going to tap the Mute button. Notice when it mutes, there's a subtle ramp to the audio. And there's an animation in the volume slider.

And then in this example, there's an interactive dismissal. And notice that the audio gracefully fades with the user's dismissal. And then when the user cancels the dismissal, the audio comes back up.

So those are two small things, but they illustrate the level of attention to detail that we're able to give to AVPlayerViewController and that your users will get for free if you're able to adopt it in your app.

You get a whole bunch of other things. And the question that you might ask is how do you get all of this? And let's take another look at code.

So, this is basically the same thing that I showed earlier in the talk. The only addition is this external metadata. This line allows you to supplement your metadata. And it really is that easy to get started.

But it's worth talking about why it's possible for you to do so little on the UI side and to get such a good experience. And so, for that I want to talk about a couple of best practices here. So, when you use AVPlayerViewController for a full screen presentation you should always present it modally, not as a child view controller of some other view controller. And sometimes you may have done that because you wanted to learn about presentation state, which you can now do with the callbacks that I discussed earlier.

Presenting modally lets us handle things like status bar visibility. Let's us optimize the screen's display mode. It lets us handle video gravity, video zoom mode, interactive dismissals. And when you are presenting modally full screen, you should use the default modal presentation style. As you may know, in iOS 13, the new default modal presentation style is automatic. And what that resolves to for AVPlayerViewController is a full screen presentation.

And there's two types of full screen presentations: Full screen and over full screen. The reason why you want to use the default is because when your view controller is presented, the presenting view can be removed by UIKit. This allows, because your view is covering the newly presented view hierarchy, removing the presenting view is an optimization during rotation. You don't want to do layout when your view's not even visible. But it also allows things like landscape playback if your app is otherwise portrait only or this view controller hierarchy is otherwise portrait only.

It's also best just to leave the video gravity property alone for full screen playback. That allows us to do automatic behaviors like I was illustrating before. If you set the video gravity, we'll give you exactly what you want. We'll zoom the video or unzoom the video. But once you do that, the automatic behavior, we can't do the automatic behavior anymore. So your users generally would prefer to have the automatic behavior. And of course, use the new delegate methods to track the full screen presentation state.

A couple of tips related to AVFoundation best practices here. What this one is about, the way I think about it, is if you're starting a video, say halfway into the video, seek the player item to the resume time before setting it on the player. And make sure you're going to have a playing player that it's already playing. And then provide that to AVPlayerViewController. And this allows the UI to look correct. And more importantly, it avoids loading content that isn't going to end up being played.

You should also always observe AVPlayer's and AVPlayerItem status property.

Though you can use KVO to observe them. When they fail, when the property status becomes failed, there's an easy path to recovery in many cases. Check the error property. See if media services were reset. And if so, just rebuild your player, player item, your AVFoundation object ref and set it back on the PlayerViewController and everything should be fine. And this property gives you an optimized video rendering when you're mirroring. And of course, always configure your audio session for playback.

So, when we talk about embedding AVPlayerViewController inline, there's a little bit more to consider.

And we've already talked about the full screen, new full screen presentation state callbacks in iOS 13.

So I won't go over those again. But, one thing I want to touch on is if you've used AVPlayerViewController in the past, when we enter full screen from an embedded inline presentation, the modal transition style has been in over full screen style, which means that your view, your presenting view, isn't removed. If your app links against iOS 13, the default presentation style now will be full screen. Which means that the presenting view might be removed. And that has all the benefits that I talked about earlier.

So, ideally you'll be able to leave this modal presentation style as full screen. But if you don't, it's okay. You can still set it to over full screen. But the recommendation is to just stick with the default.

Another thing to consider is when you have an embedded presentation, your video, you may want to zoom the video so that you don't have black bars on top or on bottom. You also might want to have things like a corner curve to the video's view port. And also, you may want to set the background color of the view. You can do all that to AVPlayerViewController's view or to the video gravity when it's embedded inline. And we'll use that for the inline part of the presentation. If the user decides to go full screen, then we'll do the right thing there. So, in contrast to, if you're just presenting AVPlayerViewController full screen, it's okay to -- you can change these properties for the inline portion of the presentation. No problem at all.

We also have a couple of properties that provide automatic behavior when you have an inline presentation. And these allow you to do things like, if the user taps a big play button, the PlayerViewController will -- the content of the PlayerViewController will automatically enter full screen.

And similarly, at the end of the video, will exit full screen. And it's really best for us to handle those things on your behalf. Because for example, let's say that a user is scrubbing to the end of a video.

Well, the time's going to be the end of the video, but you don't want to dismiss the video at that point. Because they're probably just seeking around. We keep track of what the user is doing. And we'll only exit if it's appropriate.

Of course, you're always going to want to adopt UIViewController containment API. And it looks like this.

One other thing to keep in mind is if you have a lot of videos that you're scrolling through, you may find that having a poster image is a good idea as a way to show a frame of the video without having to have the video fully loaded. And that can take a little bit of time. And so you use AVPlayerViewController's contentOverlayView for this. But then you want to know when is the video -- when is the first frame of video ready so that you can remove the content that you just put in the overlay view.

You do that by observing isReadyForDisplay on AVPlayerViewController. It's a property that you can observe using KVO. And this is what it looks like to observe it. You always want to take the initial value because it could be the video frame is already rendered.

I want to finish by talking a little bit about picture-in-picture, which is super easy to add to your app. AVPlayerViewController takes care of most things for you. You do have to configure your application to support picture-in-picture. And it looks like this. Just click a check box in XCode.

One thing, this applies whether you're using AVPictureInPictureController or AVPlayerViewController. When a user swipes up to go home in iPad, picture-in-picture may automatically start. And that means when your app is entering background, you should not pause any playing video. Because picture-in-picture may be starting. You don't want to pause the very video that your user is trying to keep on watching. So, don't pause. And AVFoundation will take care of this for you. But if you absolutely must pause, you should wait for the application's background state, application state, or the window scene state to reach background. And this is how you do that.

You can track state about whether you're in picture-picture or leaving picture-picture using the AVPlayerViewControllerDelegate.

And always toggle PictureInPicture Playback.

Be prepared for your view controller to dismiss when picture-in-picture is starting. And unlike in the full screen case, we will prevent deallocation of AVPlayerViewController while picture-in-picture is active. But if the view controller's been dismissed, you're going to want to restore the UI when the user wants to go back to your app. And so, it's really easy to do that. This is the callback you're going to get AVPlayerViewControllerDelegate. And all you have to do here is get your UI back into a state that works, that is what you want. In this case we're just presenting the view controller again. And then when you're ready, call the completion handler that is passed into the callback. Let us know that you're ready to go. And we will animate back in place. You need to do it fairly quickly, otherwise the picture-in-picture window might just go oop! And no more picture-in-picture for your user. And they're not able to get back to your app.

So, that's a look at what's new in best practices for AVPlayerViewController in iOS. There's sample code illustrating all these concepts at our session website. And now, I'd like to hand things over to my colleague, Dan Wright.

All right. Let's talk about video playback on TV.

On tvOS AVPlayerViewController supports full screen interactive playback with standard controls for navigation, information, and settings.

Built in support for advanced Siri features such as "what did she say?" Publication of Now Playing info. Interstitials such as for ads. Content proposals, and more.

So, what's new for tvOS 13? Since last year we've updated the appearance of the controls. And starting with tvOS 12.3, introduced fine precision scrubbing.

Today, we're introducing custom interactive overlays, channel flipping for live streams, and automatic enforcement of parental content restrictions.

Fine precision scrubbing makes it easier to use your Siri remote to find the precise moment in the video that you're looking for.

PlayerViewController has always supported -- the scrubbing in PlayerViewController has always made it easy to navigate quickly across a wide range of your video. But now fine scrubbing makes it easier to find a very precise moment, like within about a second, as you're navigating.

And of course, the info views have a new appearance to match the TV app.

Custom interactive overlays support application controls. A custom overlay is normally hidden.

A hint onscreen guides the user to swipe up, which presents your overlay. The overlay is fully interactive, and it can contain buttons, collections, or any other interactive elements that you like as defined by your view controller. And then you should, for persistent non-interactive elements like a channel logo, for example, you should continue to use the contentOverlayView too. Let's see what custom overlays look like. When the video begins, you have a hint at the bottom to guide them to swipe up for more. And when they swipe up, they get your overlay. Just like that. They can dismiss by swiping down or by clicking the Menu button.

So, your view controller is presented. You get all the usual UIViewController notifications. We handle dismissal and animations. And your users get a consistent experience across applications. So all you need to do to use it is to define your view controller with the content that you want to display. And set the customOverlayViewController property on your player view controller.

Live broadcast channel flipping.

Channel flipping is for live streams. It provides support for flipping between multiple live streams. For example, different channels. The user swipes horizontally to go to the next or the previous channels.

And a channel interstitial screen describes each channel while it loads. So let's see what channel flipping looks like. As the user swipes between channels, your channel interstitial view is shown.

Here it has a simple green background, some text, and a loading indicator. But, that's entirely within your control. You can make it look whatever you like in there, including pictures, a gradient background, your custom spinner, whatever you like. The main thing is to orient your user to where they are and where they're going.

And the one other thing I would add is that this a non-interactive screen. It's meant to guide the user to the next channel. So you shouldn't expect to have controls here. To support channel flipping, extend your AVPlayerViewControllerDelegate to implement playerViewController skipToNextChannel and skipToPreviousChannel.

All these methods do is replace the content in your player view controller with that of the new channel. You can wait until the content is ready to play if you like.

When you're done, call the completion block. Pass True if it was successful. And pass False if something failed.

But whatever you do, make sure that you call that completion block eventually, because that's what's going to dismiss the channel interstitial.

Now, there are two additional methods: nextChannelInterstitialView Controller and previousChannelInterstitialView Controller. And these are responsible for providing that view controller that displays between channels.

So, you need to instantiate your view controller, if you have one, if you need to. And populate it with information about the new channel.

And then just return it. In this example we're instantiating a new view controller every time. But in many cases, you could probably just instantiate it once and then reuse it.

tvOS has built in support for restricting access to content using a passcode, or with a device profile. Device profiles are commonly used in a classroom setting or perhaps a business setting.

In tvOS 13, AVKit provides automatic support to help your users regain control of restricted content. So, to test content restrictions, go to Settings, Restrictions.

Turn them on. Provide and confirm a passcode.

Then change the allowed content rating under Movies, and/or TV Shows.

In this case we'll change it to PG-13.

So, what do you need to do to actually support it in your app? Well, the most important thing is you need to specify the media content rating in all of your content. If your asset does not have this information, then you'll need to add it using the external metadata property of AVPlayerItem. And you're going to want to use the iTunes metadata content rating identifier.

You'll want to provide this rating at the start of playback so that AVKit can test whether this is restricted content according to the user settings.

And present the passcode to screen if necessary.

Now, if the content is restricted by passcode, the user will be prompted to enter it.

If it is restricted by a device profile, however, it could simply fail to play entirely. There may be no option for the user to override.

Now, AVPlayerViewController will request access automatically to your content when playback begins. But you can request access earlier using a new API request playback restrictions authorization.

And of course, any kind of failure here, if the user doesn't know the passcode or doesn't want to enter the passcode, or if it's a device profile that prohibits it, the player view controller will be automatically dismissed. And finally, if you have a custom playback user interface, you can also request access using the aforementioned API to restricted content. Now let's take a look at what the user sees when attempting to play restricted content. Now, the video's going to load. And as soon as it's ready to play and the metadata's available, the passcode screen is going to immediately appear. And as soon as the user enters the passcode, the passcode screen is dismissed and the video begins playing. So let's take a look at how to provide the media content rating.

This metadata is sometimes part of your assets. But usually it is not.

So you will need to provide it via the external metadata property of the AVPlayerItem. We've seen this code before in previous presentations. But we just have a helper function here that takes our rating in the form of a string and returns and AVMetadataItem.

The value is that string. The extended language tag is set to und for Undefined because it's not any particular language.

Your identifier is the iTunesMetadataContent Rating.

And then finally, once we've created it, in this case with a rating of PG-13, we add it to the external metadata property of the player item. Now, AVPlayerViewController will prompt automatically, if necessary. But let's say you want to prompt earlier before you present the player view controller, or perhaps before earlier content. You can do this using the requestPlayback Restrictions Authorization API on AVPlayerItem. The closure will be called when the request is complete to indicate whether there the request succeeded or failed.

And if it fails, the error provides information about why it failed. If it succeeded, you should proceed with playback. Otherwise, you should cancel, dismiss, go back to your menu screen or whatever. And don't try to play the content again unless the use explicitly requests it.

All right. Let's do a brief demo and get a closer look at some of this.

All right. So we have our little sample app.

These videos may be familiar to some of you who have done testing with HLS.

Here's our swipe up. Swipe for more hints at the bottom. And if we swipe up, we get our custom overlay. And see that it actually works and everything. And I can just swipe down to get rid of it. And what else do we have here? Well, if I press and keep my finger down, we see the Hints on the left and the right sides of the screens. Those are the arrows to guide the user to channel flipping. And if I just swipe -- see what's going on next door.

Using some great design patterns, including -- Oh yeah, it's the bottom part is using a list to lay out -- In the last month -- Let's see. What else do we have? Is that it? Oh, what's going on over here? All right.

All right. So we have our little sample app.

I passed me. Let's go back to -- let's see -- back to the -- Here's our Swipe up bar for more hints at the bottom.

And if we swipe up, we get our custom overlay and see if it actually works -- Let's pause.

All right. So let's take a moment to talk about some best practices on tvOS now.

Some of you already have controls in your apps that are revealed by swipe up. Often, it's Up Next list.

And typically, you're doing this by installing your own custom swipe up gesture recognizer on the player view or a super view. And then presenting your view controller. Well, if you're doing that, we want to encourage you to migrate to using a custom overlay instead.

What that gets you is much better compatibility in the future -- hint. And a better and more consistent experience for your users across applications. If you're using a UI tap gesture recognizer on the Menu button to detect dismissal of your player view controller, or in some cases to provide the dismissal animation, you should use the delegate to dismissal notifications instead. We introduced those in tvOS 11. And they should give you all the information that you need. This is important because it's no longer the case that the only way that your play view controller can be dismissed is by the user hitting the Menu button. It could now be automatically dismissed.

I've said this one before, but showsPlaybackControls should not be abused to try to force immediate change in the visibility of the controls. The purpose of showsPlaybackControls is to configure the player view controller to indicate whether you want the user to have access to the playback controls at all.

Provide media content ratings for all of your content.

And of course, test with parental content restrictions enabled.

Most of you will have no issues, but you might find that there's something that you're doing that doesn't work very well when a passcode stream comes up. For example, if you display a hint for a few seconds at the start of playback, you would want to not do that while the passcode is covering it or the user won't see it.

So in summary, adopt AVPlayerViewController across platform. You get great functionality on iOS and on tvOS for a very low cost.

And a wide range of compatibility for remotes and other features.

Use AVPlayerViewControllerDelegate notifications to track presentation state, as seen today on iOS, and to handle dismissal, as on tvOS. Observe the player item status and handle errors.

Use external metadata, now on both iOS and tvOS in addition to some of these uses we've seen today for parental content restrictions and in the info panel. External metadata is also used to publish Now Playing info, which means that if you use external metadata on both platforms, you can now remove all of your Now Playing info support.

If you have a custom overlay on a swipe up action, migrate it -- I'm sorry -- if you swipe up action presenting an overlay, migrate to custom overlays.

And finally, support parental content restrictions.

So, for more information, including sample code for this session, visit our session page at the URL on the screen. And come with your questions and your code to the AVKit Lab at 2:00 p.m. today or to the AVFoundation lab at 4:00. Thank you, and enjoy the rest of conferences. [ Applause ]