Discover generated subtitles and subtitle styles

Hi, I'm James and I'm an engineer on the AVFoundation team. I love to travel, so today, I'll take you on a trip to explore a couple of features related to subtitles. People use subtitles for many reasons. Subtitles are vital for people who are deaf or hard of hearing, or who have other accessibility needs. They need subtitles to understand the content. Some people read subtitles to assist in understanding the spoken dialogue. I use subtitles when I physically can't hear the audio, like when I'm in a busy airport.

Apple AI-generated subtitles can be created live, locally on the device as the media plays. In this video, first, I'll show you how the Apple AI-generated subtitles appear in your app.

Afterwards, I'll show the subtitle style preview feature that lets people customize subtitles while playing video.

Next, I'll show how media assets are authored.

The media creation journey usually begins with filming and editing the Video and Audio content.

Then, subtitles, text that represents spoken words in the audio, are created manually. Later in this video, I'll refer to such subtitles as Authored subtitles. The content author can create multiple subtitles, each for one particular language. Also, they can create multiple audio languages. The final media contains the Video, Audio and Authored subtitles all together.

Even content with multiple audio and subtitle languages might not have a language the viewer understands.

People can have their device create generated subtitles to help fill the gap when the original content doesn't have the subtitle language they need.

This enables more people to access and experience the content.

Next, this is how generated subtitles are created.

There are two use cases for generated subtitles. In the first use case, subtitles are generated from audio.

The source audio goes into the on-device Speech-To-Text model and subtitles are created.

This is called Speech transcription.

Second, for language translation, subtitles are generated from other subtitles. The source subtitles, English in this case, go into the on-device translation model. New subtitles in a different language are created, Italian, for example. This is called Language translation. Generated subtitles provide additional languages. The authored subtitles are preferred and remain unchanged.

The good news is that you don't need to implement anything to turn on generated subtitles.

They're available automatically during video playback.

Generated subtitles are provided for many playback scenarios. Subtitles can be generated for HTTP live streaming content, including Live streams, like TV channels. It also includes Video on demand movies and shows, including travel videos and live events, like sports.

File-based content is also supported, like app-bundled videos or downloaded media.

Several content types are supported. Professional content, like movies and series, is supported.

So is customer-created content, like camera capture from the iPhone and social media videos.

Now, I'll show you the supported devices and languages. Generated subtitles are available on multiple devices. This table shows the supported devices and languages.

Starting in iOS and macOS 27, English subtitles can be generated from English audio. This is also supported on tvOS and visionOS 27. Also, multiple subtitle languages can be generated from English subtitles on iOS and macOS. Now that you know about generated subtitles, I'll talk about how to present them in your app.

It's important to provide subtitle selection UI during video playback.

Here are the options for your app.

This is the AVPlayerViewController UI on iOS. It fully implements subtitle selection and player controls. You don't need to do anything extra. AVPlayerView on macOS provides similar functionality.

This is AVLegibleMediaOptionsMenuController. It presents subtitle selection controls and implements the behavior. It's a good choice when you want to add subtitle selection UI to your existing player UI, since it does not provide player controls.

Or, you can implement custom controls for media selection to match the style of other controls in your app. Like the ones I created here in my app.

Generated subtitles make the content in your app more accessible. More people can understand and enjoy the content.

Now I'll show you another subtitle feature that makes your app even more accessible.

Subtitles help make content accessible, but the presentation of the subtitles is also important. The Settings app has let people select and change subtitle and caption styling for many years.

There are a few built-in styles, and people can create custom styles to fit their needs. I've created a custom style called Bold Yellow. It has yellow text with a little extra border to make it easier for me to read. I can select the style in the Settings app, but it would be easier, and more accessible, to change the style while watching a video. That's exactly what the subtitle style preview does.

I'm watching a video and I have the Style menu open. I have the same styles available that were in the Settings app.

The subtitle style can now be changed right from the menu where you select subtitles during video playback. Not only that, but a preview of the style is shown to make selection easier.

This is what my Bold Yellow style looks like in a video.

There are several ways to implement the subtitle style preview feature in your app.

First, this is AVPlayerViewController on iOS. It fully implements the subtitle style preview and player controls. AVPlayerView on macOS provides similar functionality.

This is AVLegibleMediaOptionsMenuController. It presents subtitle style preview controls and implements the behavior. It's a good choice when you want to add the subtitle preview to your existing player UI.

Next, AVPlayerLayer has an API to show the style preview. I'll show you how to implement it in a moment. Also, AVCaptionRenderer can provide the style preview, but you are responsible for rendering it. Now, here's the AVPlayerLayer implementation. Each subtitle style in the system is assigned a profile ID.

First, fetch all of the styles by their profile IDs. Populate your UI with the names for the styles.

When a style is selected, show the stylized preview. New subtitles are shown using the specified style. Any existing subtitles are automatically hidden so they don't interfere with the preview.

Pass nil for the text parameter. In that case, localized system text is shown.

Use the position parameter to avoid any UI controls. This is an offset from the default location of the preview text.

Call this function again to show a different style. You can call it as many times as needed.

Stop the preview when selection is done. This removes the preview text and restores any existing subtitles that are active.

Set the chosen style. It will be used for all subtitles on the system.

You can build this feature in your app to let people quickly choose subtitles that are easier to read while watching a video. Now, I will show you both of these subtitle features together. I'm planning a camping trip to Italy. I'm going to watch this camping video for some inspiration.

It's has English subtitles, but I want Italian subtitles, so I can brush up on my Italian.

To change the language, I'll open the Subtitles menu, then click Language.

Several subtitles are available. Some are authored and some generated. The generated options are marked with a sparkle symbol and the word Translated. I'll pick the Italian generated subtitles.

Now I have Italian subtitles in my video.

I'll change the style to make them easier to read. First, I'll open the Subtitles menu, then select Style. I'll try Large Text. The existing subtitles have been replaced with a placeholder message, in Italian, using the Large Text styling.

The text is larger, but let me try my custom style.

Oh, I like that.

I'll dismiss the menu.

The subtitles are now using my custom style. This trip is going to be amazing. Before I go, here are your next steps.

Now that you know about the generated subtitles feature, I encourage you to go explore it. For example, watch some travel videos and turn on generated subtitles.

Also, make sure your app has UI to select subtitles. Implement the subtitle style preview too. People appreciate this accessibility feature when they need to change the style of their subtitles. Thank you for watching. I have to go catch a plane to Italy. Ciao ciao.