Meet Group Activities

♪ Bass music playing ♪ ♪ Pierre de Filippis: Hi there. My name is Pierre.

I’m a manager on our Communication Applications team, and I’ll later be joined by my colleague Bhaskar.

It’s my pleasure to welcome you to “Meet Group Activities.” I think we can all recognize that this past year, finding ways to stay connected has felt more essential than ever.

This is incredibly important to us at Apple, and we wanted to give our users meaningful new ways to connect.

As a result, in iOS 15, we’re creating a new way to enjoy activities together that makes you feel like you’re in the same room as the people sharing with you.

And we call it "SharePlay." SharePlay is powered by the Group Activities framework, and in this session, we’re going to talk about how you can adopt it in your application.

Before we get into some of those details, let me tell you a bit more about the key components of SharePlay.

Let’s start with communication.

We think dynamic and natural communication is a critically important part of a meaningful shared experience, so we’ve built SharePlay right into FaceTime and Messages.

We are continually blown away by how central these apps have become in people’s lives and relationships, especially over the last year.

And people don’t just use FaceTime and Messages to connect with that friend from 30 years ago once in a while.

Instead, our users spend a significant amount of time communicating with the people that are closest to them; their closest friends, and their closest family.

Those are the kind of people you would invite into your living room to watch a movie, for example, and that’s exactly what we want to facilitate with SharePlay.

Let me give you an idea of how this works.

It all starts with the concept of a session.

Before a group starts an activity, they need to join a session so that everyone is together and ready to share.

Once users are in a session, they can communicate via text, audio, and video in Messages and FaceTime just like they are used to.

Switching between these modes of communication is dynamic and fluid, so they can easily choose a way to communicate that matches the moment.

We also give users a consistent system-wide way to manage their session.

Users can invite new people on the fly or leave the session entirely.

Once you are in a session, users can navigate the entire system and drop in on any app.

As a developer, you get it at all for free with Group Activities.

So that’s communication.

The next area we think is critical is platform experience.

We want this experience to be simple, consistent, and available across our system and devices.

With Group Activities, we’re building a unified experience that works across iOS, iPadOS, macOS, where Group Activities is available even for your website with WebKit.

It also works beautifully on AppleTV, so you can enjoy it on the big screen.

All of your devices are aware when you are in a session, so you can join from any device, and you can use multiple devices seamlessly.

Finally, we also designed it to deliver great audio to Bluetooth devices, including AirPods.

As part of platform experience, we’re doing something special for content playback.

While we’re hoping you take full advantage of the expressiveness of the API to create many different types of SharePlay activities, we created some really cool enhancements that make shared media experiences effortless to build.

The core of any media experience is the play button; it’s the moment when your users decide what content they want to spend their time on.

Our goal is for every play button throughout the system to work with SharePlay.

We want people to feel that any piece of media in your app is ready to kick off a shared experience whenever they’re on FaceTime with their friends, which is where you come in.

We’re offering new APIs designed to fit right into your existing code paths.

By adopting Group Activities, your play buttons can now start these shared media experiences whenever a group conversation is active.

And we’ll use smarts over time to adapt to user behaviors, making shared experiences effortless.

Next is time-synced playback.

We think it’s super important that people are laughing together, smiling together, and reacting to content all at the same time.

SharePlay handles keeping playback time synced at the platform level, so you can simply adopt and not have to worry about rolling your own.

Now, this was a challenging problem to solve when it came to consuming content with SharePlay.

We wanted to allow people to experience the exact same moments with one another, even if they were halfway across the world.

Imagine seeing and hearing your friends laugh at something that hasn’t happened yet.

Spoiler alert! So we built an entirely new playback-sync protocol with deep integration into the AVFoundation stack on the device.

That means someone hits play, and everyone in the group immediately starts playback at the same time.

Jump to a favorite scene and everyone else sees it too, allowing people to experience moments together in perfect sync, as if they were in the same room.

The magic behind this playback synchronization means we won’t retransmit your media in any way.

Everyone will get your full-fidelity video because it’s playing right from your app and streaming from your servers as it always does.

And by ensuring that all viewers are watching the highest-quality video, they’ll get an immersive social-viewing experience that never compromises the quality of your content.

We’re also doing some groundbreaking work to make communication feel natural during playback.

With smart volume, when people speak up during playback, we’ll automatically duck the audio of the content and bring it back up when appropriate.

So people can communicate via voice, video, and text naturally, just like they would if they were sitting in the very same room.

And of course, it works beautifully with Picture in Picture, so the user can take your content along with them as they multitask on the device.

So that’s a brief overview of what we’re doing on platform.

And finally, content. And that’s where you come in.

We want users to be able to enjoy the very best content that you've made available in your apps.

When people are on a FaceTime call, they will expect to be able to go into your app and share your content.

SharePlay is going to allow you to extend the touchpoints of your product and increase the times where your app is relevant to your users.

And this will happen very organically.

Existing users will evangelize for your app as they are using SharePlay with other people.

Now that I’ve given you a brief overview of some of the key components of SharePlay, let’s turn our attention to Group Activities, a core concept of the framework.

A Group Activity is an object representing something that can be shared and enjoyed with the people in the FaceTime call using SharePlay.

Let me walk you through how a user would start a Group Activity in your app.

While on a call, users can navigate to your app, and if you have adopted Group Activities, they will be notified that the app supports SharePlay.

To configure the activity you want to share, you first need to create an object that implements the GroupActivity protocol.

Once you’ve configured your activity, you call the prepareForActivation API to start sharing that activity.

This API will offer the user the option to share that activity with everyone in the FaceTime call or just keep it local.

If they decide to share with the group, Group Activities will notify you and let you join the GroupSession object.

Once users have joined the Group Session, their video will be kept in sync with the group, even when they play, pause, or seek.

Whenever those events happen, Group Activities will automatically notify the user as shown here.

And finally, once users are done, they can choose to end the activity for themselves or for the whole group.

It’s as simple as that! Now, let me hand it over to Bhaskar who's going to walk you through the rest of this presentation.

Bhaskar Sarma: Thank you, Pierre.

Hi, my name is Bhaskar, and I am an engineer on the FaceTime team.

Now let’s talk about some of the high-level concepts and architecture behind this new framework.

GroupActivities is a Swift-native framework that allows you to create shared experiences between users of your application when they are on a FaceTime call.

Additionally, the framework also has a tight integration with AVFoundation, which allows you to easily create shared video and audio playback experiences.

Before we go over the high-level concepts behind the API, I would like to talk about two key parts of the API, and these are the GroupActivity and GroupSession.

GroupActivity is what your app uses to define the shared experience.

It holds the information that your application needs for the shared experience.

For example, if you are doing a shared audio or video playback experience, it might hold the URL of the content that is being played.

Or your application might provide a custom shared experience.

For example, you could be providing a draw-together experience.

In that case, this would hold information about what the users are drawing.

GroupSession is basically a representation of the group taking part in the shared experience.

It provides access to things like the participants in the group.

And there are additional APIs in the framework that you can use along with the GroupSession to send and receive data across devices.

Please note the GroupSession is not meant to exchange large amounts of data.

For example, if you are creating a shared listening experience, the GroupSession is not used to exchange the contents of the song.

Instead, this channel is used by AVFoundation to keep content in sync by exchanging the play, pause, or seek commands.

Also, the communication medium provided by GroupSession is end-to-end encrypted.

This means no one other than your application on the device can read the data that is exchanged over this channel.

Here I have two phones on the screen.

The one on the left is my phone, and the one on the right is my teammate Pierre’s phone.

And we are on a FaceTime call.

Also, we have a sample application called "Awesome App" on the screen, and the app is ready to start a shared experience between me and Pierre.

So in this case, the application on the device on the left is going to initiate the activity.

The first thing the application needs to do is create an object that conforms to the GroupActivity protocol.

In the example shown here, it is the AwesomeActivity object that conforms to GroupActivity.

As I had mentioned earlier, this object conforming to the GroupActivity protocol contains information about the shared activity.

For example, if you are creating a shared playback experience, it will have information about what content to play.

Or if you are creating a custom experience -- like drawing something together -- it will have information about what to draw.

Once the app has created an activity, the next thing it needs to do is call prepareForActivation on the activity.

This causes the user to see a prompt asking for permission if they want to start an activity.

We would like to take user consent before starting an activity as we do not want to cause surprise to them.

Hence, this step is essential.

And lastly, after the user has granted permission to start the activity, your app needs to call activate on the activity object.

This tells the system that your app wants to start a shared experience.

Next, let’s talk about observing sessions.

Here we have the same app that we were looking at earlier.

The application at this point has called activate on the activity object.

Your application then needs to iterate over incoming sessions via an AsyncSequence on the GroupSession class.

And then, when there is a session, the application gets handed the GroupSession object for the shared experience.

Please note that this is the same step your application would take whether it was the initiating app or on the remote device receiving the session.

To go into more details about initiating and observing group sessions, I would recommend you watch this WWDC session.

Now that the application has been handed a session, it needs to set itself up before joining the session.

Setting up the application before joining could mean different things based on the specific use case of the application.

For example, if you are creating a custom experience -- like drawing something together -- this could mean loading up the assets needed for the shared experience before your user can join the experience.

However, if your application is providing a shared media-playback experience, then it would involve something like this.

Your application will hook up the AVPlaybackCoordinator of your AVPlayer to the GroupSession so that its associated AVPlayer can sync content over the communication channel provided by the framework.

Keep in mind, this syncing support is not just limited to AVPlayer, but you can use any other custom video player you have today and still get support for syncing via an AVDelegating PlaybackCoordinator.

And finally, once you are done setting up your application, your application calls join on the GroupSession.

Once join is called, the system sets up the end-to-end encrypted channels between the apps running across different devices.

At this point, your application is ready to sync data and let your users take part in the shared experience.

If you're creating a custom experience, your application can use this channel to exchange data to keep your users in sync.

This channel is also used by AVFoundation to keep media playback in sync by exchanging playback state when the user hits play, pause, skip, et cetera.

Note that this channel is not meant to be used to exchange large amounts of data.

Instead, use it to exchange information to keep your users in sync.

Now your application is in a session, and your users are enjoying the shared experience.

There is one more thing you can do to enrich the experience for your users, and that is post events using the framework.

Events allow your users to know what is going on in the shared experience.

For example, you can post events to let your users know when someone plays, pauses, or skips a track.

Posting an event causes the system to present a notice telling your users about the event.

In our current API, we only let you post events for media playback experiences.

You get this for free if you use an AVPlayer or use an AVDelegating PlaybackCoordinator.

However, if you are not using either of these, your app can still post events using the framework.

To learn more about providing your own custom experience and about posting events, I would recommend you watch this WWDC session.

So to wrap up, Group Activities is a brand new Swift-native framework that allows you to create shared experiences over FaceTime.

The framework is cross-platform and is available on iOS, macOS, iPadOS, and tvOS.

It has tight integration with AVFoundation that allows your app to provide shared media playback experiences.

And finally, it also has support for playback syncing over the web on macOS.

And with that, I would like to thank you for watching our session.

And have a wonderful WWDC21! ♪