Master Picture in Picture on tvOS

Hello and welcome to WWDC.

Hi, and welcome. My name is Jad Osseiran, and I'm a software engineer on tvOS.

In this session, we'll begin with a high-level overview of tvOS Picture in Picture.

We will then cover what it takes to implement Picture in Picture in your applications, and I will then hand it off to Dan, who will talk about how you can make your existing applications work great with Picture in Picture.

Finally, Dan will run us through a demo that covers what we've learned in this session.

Okay. So let's get started and have a look at an overview of tvOS Picture in Picture, or, more succinctly, PiP.

Here, we have an application with two videos in its browse view. Let's see how we can put these videos in Picture in Picture, and even swap between them. First, let's bring a video full screen. By bringing up the playback UI, we can select the "start PiP" button, bringing this video into Picture in Picture. Unique to tvOS, we can start the other video and have them play simultaneously side by side.

At any time, the playback UI can be brought up to swap the full-screen video with the content in the PiP. If we now go back to our browse view...

we can see that the video in PiP is the second video that we played. We have effectively PiP-ed both videos and swapped them.

So we have seen that tvOS PiP brings a familiar experience from iPadOS and augments it by adding simultaneous video playback. So how can your application take advantage of this? To find out, let's dive into how you can implement Picture in Picture in tvOS. The first step is to set up your Xcode project. Before you write any code, you will need to add the Picture in Picture capability to your application in Xcode 12. To do so, you select Background Modes in the Add Capabilities window.

It is then as easy as checking the Picture in Picture box in the Signing and Capabilities tab of your project.

Next up, you will need to configure your audio session by setting your application's AV-Audio-Category to "playback." Here is what this code might look like in your application. And that's it for project setup. If you have previously implemented PiP on iPadOS, these steps should be very familiar to you, as they are the same across our platforms. The second step in implementing tvOS PiP is adapting your playback UI. So let's see what that entails for the standard playback UI.

Our standard playback UI, or AV-Player-View-Controller, will display the right UI affordances for PiP. This happens automatically if you've configured your project following the setup steps we've just discussed.

To help you respond to the PiP's life cycle, your application will make use of AV-Player-View-Controller's delegate methods, which are now in tvOS.

If this API is new to you, or you would benefit from a quick refresher, here are some of the delegate methods we will be seeing again in this session. These methods cover the life cycle for starting PiP, stopping PiP, and restoring PiP. Now I'd like to call out the restore. In this method, your application is given the opportunity to restore its UI so that it can accommodate the player coming out of Picture in Picture. When implementing this method, keep in mind that a fast restoration is ideal, and that avoiding animations is a great way to ensure the restoration is quick for your user. If your application takes too long, the restoring player will be terminated, leading to a poor user experience.

Getting started with a standard playback UI is straightforward and very familiar with the existing iPadOS implementation.

Later in this talk, Dan will go into further details on how you can get the most out of PiP with AV-Player-View-Controller. However, we recognize that some of your applications require a custom playback UI. If this is true of your application, then your second step in implementing tvOS PiP will be different than what we've just covered. So let's explore that now.

AV-Picture-In-Picture-Controller is now available on tvOS. And just like on iPadOS and iOS, your application will use it to control the PiP.

AV-Picture-In-Picture-Controller now has a new tvOS API to help you manage your custom UI.

So let's have a look at that API.

Your app will use can-Stop-Picture-In-Picture to help you display the appropriate affordances and provide the correct behavior in your custom playback UI.

So let's dig into this a little more.

If can-Stop-Picture-In-Picture is false for your full-screen player, it indicates to your application that your custom playback UI should display a start-PiP affordance.

When your users select the start PiP affordance, your app should call start-Picture-In-Picture under managing AV-Picture-In-Picture-Controller to programmatically start PiP.

Conversely, when can-Stop-Picture-In-Picture is true, your application should display UI affordances for swap and stop in your custom playback UI.

It is worth noting that there is no special call to initiate a swap. Your application should call start-Picture-In-Picture when starting a swap, and the system will take care of the rest. If we think about this from the point of view of the full-screen player, it is starting PiP in both a simple start and a swap case, making start-Picture-In-Picture the appropriate method in both scenarios.

To stop an existing PiP from your full-screen player, your application should call stop-Picture-In-Picture on the AV-Picture-In-Picture-Controller managing that full-screen player.

It is important that your app only call stop-Picture-In-Picture when the user has selected your stop affordance.

Can-Stop-Picture-In-Picture can change at any time. For example, a user might stop the PiP from the PiP window itself. As such, your UI must always be up to date. To do so, your application will need to observe can-Stop-Picture-In-Picture for changes and update your UI accordingly.

Here's a sample of how that might look in your code. We strongly encourage you to read the latest Human Interface Guidelines, which goes into detail about how to best display these affordances. Next up, let's talk about how tvOS 14 allows publishing to multiple Now Playing info centers. When two videos are playing simultaneously, their respective Now Playing information should also be updated.

You can do this by tying AV-Players to an MP-Now-Playing-Session. Your application can have multiple MP-Now-Playing-Sessions, and in the case of PiP, you would have a Now Playing session for your PiP content as well as one for your full-screen content. In fact, in order to use PiP on tvOS, the same player used in AV-Picture-In-Picture-Controller must be tied to an MP-Now-Playing-Session. So let's have a look at what that means in code.

Here is a snippet of what a custom playback UI controller might look like.

We see that the player used to create the AV-Picture-In-Picture-Controller is also used in the MP-Now-Playing-Session initializer.

Later on, and when appropriate, you can set the session's Now Playing info center and make it active. This tells MediaPlayer to start publishing this information on your behalf. Before we move on, there are a couple of things to note when using MP-Now-Playing-Session. The first thing to note is that updating to MP-Now-Playing-Session will make the system ignore updates from the default MP-Now-Playing-Info-Center. As such, you should make sure that when you implement PiP, you switch over to MP-Now-Playing-Session across your whole application. Because there are now multiple Now Playing info centers, the system will decide which information to display in the Now Playing system UI.

It is important, however, to keep publishing Now Playing information, even if your session is not currently shown in the system UI.

Your app will want to do this, as the system may choose to display your session at any moment.

So far, we have covered setup and the two different ways you can integrate your playback UI with PiP.

The last thing we want to cover on implementing tvOS PiP is the life cycle your application should expect when performing a swap.

The first scenario is swapping within your own application. If we imagine that these videos come from your application, let's look at the lifecycle between your sports PiP content and your full-screen animated movie.

Here's a snippet of what a player delegate object would look like.

First, your application will see the will-Start delegate callback for the player that is going into PiP. Next, your stopping player will ask your application to restore a UI to accommodate it coming out of PiP. Remember, restoring quickly is essential here. Once you have restored your UI, you will get a will-Stop callback for the stopping player. And finally, your players will get their respective did-Start and did-Stop delegate callbacks once the swap has completed. The second scenario we will cover is swapping across applications. Here, your application can either start Picture in Picture, swapping with another application's PiP, or your application can be the one that is bringing its Picture in Picture full-screen. So let's first cover the case of your application starting PiP.

Let's now imagine that your application is providing the full-screen animated movie. In this scenario, the PiP sports content is provided by another application.

Now when we swap, your content is going into PiP.

So taking our player delegate from before, let's focus on the delegate callbacks of interest.

You will get a will-Start callback and as you might expect, a matching did-Start once the animation is complete.

Your application will then get backgrounded. This is because as part of the swap, another application became the foreground app to play their full-screen content. If you have implemented Picture in Picture in iPadOS, this may look familiar to you. It is the same sequence you would see when your application is automatically put into PiP when the user returns to the home screen.

Lastly, let's cover the cross-app case, where your application brings its PiP full-screen.

For this last example, let's imagine that your application now owns the content in the PiP, and that another application is playing full-screen.

During a swap, your PiP content will take over the full screen.

So let's again follow this in our play delegate example.

First your application is brought to the foreground.

Then you will get the opportunity to quickly restore your UI.

It is then told that Picture in Picture will stop.

And finally, once the animation is complete, you will be told that Picture in Picture did stop for your player. Okay. We've just covered what it took to implement Picture in Picture from scratch. After configuring your projects, we explored the two approaches to integrating your playback UI with PiP. If your application uses AV-Player-View-Controller, the heavy lifting is already done for you. However, if your needs require a custom playback UI, you can use AV-Picture-In-Picture-Controller. The last implementation step we covered was the expectations your app should have during a swap. By following these steps, you will find that getting tvOS Picture in Picture up and running is a breeze.

But what about your existing applications? To tell you all about that, I would like to hand it off to Dan, who will show you how you can get the most out of AV-Player-View-Controller.

Thanks, Jad. AV-Player-View-Controller provides the standard video playback experience on Apple TV, supporting a wide variety of remotes, skipping, scanning, scrubbing, Siri command, interstitial support, and more. And we encourage all tvOS apps to adopt it for video playback. For those of you who are already shipping apps with AV-Player-View-Controller, let's look at what you'll need to do to get your application working with Picture in Picture.

First, let's talk about interactive overlays.

Interactive overlays display custom application controls over a video, such as a channel guide or suggestions for further viewing.

In most apps today, users get to these overlays by swiping up on the remote. Since tvOS 13, the swipe-up gesture is reserved by AV-Player-View-Controller, and this gesture is required for users to reach the Picture in Picture controls. So, use the custom-Overlay-View-Controller property of AV-Player-View-Controller for your interactive slide-up controls. This property was introduced last year in tvOS 13. If you haven't adopted custom-Overlay-View-Controller already, you will need to do so when you move to the latest SDK, because your custom swipe-up gesture recognizer will stop working.

Users can access your custom overlay by swiping up quickly on the remote or by clicking an on-screen button when the transfer bar is visible. For an in-depth look at custom overlays, see the presentation for "Delivering Intuitive Media Playback with AVKit" from last year's WWDC. The button for the custom overlay controls appears on the left when the transport bar is visible.

Next, let's talk about Now Playing info. Now Playing info is metadata, static and dynamic, about your video, relayed to the OS, other applications, and remote devices. AVKit publishes Now Playing info on your application's behalf. So if you need to augment this metadata, use the external metadata property of AV-Player-Item rather than talking directly to the MediaPlayer framework.

In tvOS 14, the old shared Now Playing info center is no longer used. So make sure you aren't still trying to publish this info yourself when you update your app to the latest SDK.

This code illustrates using a helper function to create metadata to put in the external metadata property. Most of you are already using similar code to provide title, description, media content rating and other information to be displayed in the info panel.

As you can see here, it can also be used for non-displayed metadata such as a service identifier.

Next, let's talk about how this affects the architecture of your app. The AV-Player-View-Controller delegate receives and handles messages from AVKit about user actions and other events, including Picture in Picture. In the past, you may have had the parent or presenting view controller act as the delegate. However, the delegate must continue to exist and respond to messages during Picture in Picture. But at the same time, users can navigate throughout your application. So this means your delegate must not be part of your view hierarchy.

It should probably be a separate object that can persist while your video is PiP-ed. Let's think about what will happen when the user swaps two videos. So visually, the two videos will trade places on the screen with a smooth animation. But you'll need to do a few things to make this happen. If you're simply presenting the player view controller, then you can implement swapping by dismissing one view controller, the one going to Picture in Picture, and presenting the other. Embedding AV-Player-View-Controller in another view controller poses additional challenges. Do you want or need to recreate the parent view controller when a PiP video returns to full-screen? And think about what should happen to the navigational hierarchy when this happens.

What part of your app should the users see after they dismiss the video? Should it be what the users saw right before they started the last video? Or maybe the landing page for the video they just swapped to full-screen.

Perhaps the main screen of your app.

Depending on what you want to do, there may be additional work to perform when videos swap. Don't overlook these details. Be sure to allow time in your process to properly design and implement the right solution for your application and your customers.

If you have a custom playback UI, you may currently be pausing playback when your app is deactivated. But you should not do so if the video is in Picture in Picture. You do not need to do so at all if you use AV-Player-View-Controller, and we recommend that you don't. So we've talked about implementing Picture in Picture with AVKit. Let's see what it looks like.

Okay. So here we've got a video playing. Let's move this to Picture in Picture. I'll swipe up to move the focus to the PiP buttons.

And we click to go to Picture in Picture. Now let's start another video in our app.

All right. Let's swipe up again to bring up the PiP controls.

Swap and stop PiP are in the middle. We can swap the videos like so.

All right. Let's try that one more time in case you missed it. We swipe up again, and we click, and there it goes.

So let's look at the custom overlay. The button for the custom overlay is on the left.

And click, and here's our custom overlay. The PiP window, you'll notice, moves out of the way of the focus, so your users don't have to move it to a different corner of the screen to see your controls.

Now let's stop Picture in Picture. And there we go. In summary, use AV-Player-View-Controller to get the full tvOS playback experience, including Picture in Picture, with very little work in your application. If your needs require a custom playback UI, be sure to follow the Human Interface Guidelines to ensure that users are as comfortable using PiP in your application as any other on Apple TV. Do not install a custom swipe-up gesture recognizer for your own overlaid UI when you're using AV-Player-View-Controller. And be sure to allow time to adjust your application architecture to provide the best possible support for Picture in Picture. Thank you very much, and enjoy the rest of your conference.