Take your iPad apps to the next level

♪ Bass music playing ♪ ♪ Chris Donegan: Hi, I'm Chris, an engineer on UIKit.

I'll be joined later by my colleagues Anant and Mohammed.

iPad is the primary device for many people due to its power and capabilities.

In this video, you'll learn about some exciting new features you should use to take your iPad app to the next level.

First, I'll show you the latest advancements in multitasking and scenes.

Then, Anant will walk you through the powerful changes to keyboard shortcuts.

Finally, Mohammed will show you the latest enhancements to the system pointer.

Let's get right into multitasking.

iPadOS 13 introduced the ability to run multiple instances of your app's UI.

In iPadOS 15, we're improving this experience with new APIs for scene presentation as well as enhanced state restoration.

These enhancements build upon the existing UIScene infrastructure.

If you haven't adopted UIScene, check out "Introducing Multiple Windows on iPad." Let's briefly go over the key concepts.

A scene represents a single instance of your app's UI.

The structure of a scene's components is defined by a scene configuration.

At a minimum, it defines the scene's role and delegate class.

You can also provide a name, storyboard, and scene subclass.

Scene configurations can either be declared in your Info.plist, or created at runtime using the UISceneConfiguration object.

The content of a scene is represented by an NSUserActivity.

These activities are used for requesting scenes as well as for state restoration.

A scene is managed by a scene delegate.

The delegate is responsible for setting up the UI, responding to lifecycle events, as well as saving and restoring state.

Finally, a scene is tracked by a scene session.

The scene itself can be disconnected and reconnected by the system when it's in the background.

The scene session tracks the scene regardless of its connection state and persists between launches.

The session can be thought of as the representation in the system app switcher.

Each item in the switcher corresponds to a scene session.

When requesting a scene from the system, you can provide an options object for customizing the request.

iPadOS 15 has a new options subclass specifically for window scenes.

Using this subclass allows you to specify a presentation style.

Window scene presentation styles influence how the scene is presented relative to other scenes in the workspace.

There are three possible values: prominent, standard, and automatic.

This is a scene using the prominent presentation style.

It's presented modally in the current workspace with the scenes behind it dimmed.

Because it's modal, it should provide Cancel, Close, or Done buttons.

This new style can be thought of as a staging ground for new scenes.

It can be repositioned like any other scene using the new multitasking controls, and it can also be moved into the app shelf for later.

When considering if this style is appropriate for a scene, there are some guidelines to keep in mind.

First, prominent scenes should be useful on their own.

They should not be used for providing options or tools for another scene.

They should provide a Done or Close button, and they should be dedicated to specific content within your app like a document or file.

This dedicated content scope must be defined in the scene's activation conditions.

For more information on activation conditions, check out "Targeting Content with Multiple Windows." This example shows how Safari is opening a new scene using the standard style.

They are presented side by side, allowing interaction with each, and both provide the full functionality of the main scene.

You can also specify automatic as the style.

This value tells the system that it should choose the best style based on how it was requested.

iPadOS 15 not only provides a way to customize presentation using styles, it also makes it easier for people to open their content in new scenes.

On the Mac, it's common to find an Open in New Window item in context menus.

You should bring this familiar experience to your iPad app by using UIWindowScene.ActivationAction.

It's a new UIAction subclass for requesting scenes that can be used in menus, buttons, and bar button items.

To add this functionality to your app, start by initializing a UIWindowScene.ActivationAction.

It's initialized with a closure that is executed when the item is selected.

The closure should return an activation configuration with a user activity for the new scene's content.

Finally, put the action in a menu and you're good to go.

On iPad and Mac Catalyst, the menu shows an Open in New Window item that, when selected, will present a new scene.

On iPhone, the item is automatically hidden because multiple scenes are not supported.

If you would rather show another item in its place, you can provide an alternate action.

Let's update the previous example to offer one.

To do this, start by creating the alternate action.

The new action's title and image are shown when multiple windows are not available.

Then, update the initializer of the scene activation action by passing the new action as the alternate.

With an alternate action provided, the menus on iPad and Mac Catalyst still show the Open in New Window item but on iPhone, the Show Details item is visible; all done without a single condition in your code.

Adding these menu items is a clear and familiar way for people to open their content in new scenes.

But it's not the only way.

iPad is a touch-first device and it's easy to open scenes with a single gesture.

In the Notes app, pinching out on a cell opens the note in a new scene.

The scene interactively animates from the cell to its final position.

There are two ways of offering this functionality in your app.

If you're using a collection view, there's a new delegate method.

For other views, use UIWindowScene ActivationInteraction.

Both of these are only for presenting scenes with a prominent style.

To support this gesture in your collection view, implement the new delegate method named sceneActivationConfiguration ForItemAt indexPath.

It's very similar to the context menu example from earlier.

Create the user activity for the new scene's content, then return an activation configuration with that activity.

You may not want every cell to support opening a new scene.

To prevent the gesture from beginning, just return nil.

To support this gesture on other views, create a UIWindowScene .ActivationInteraction.

It's created with a closure that accepts two arguments: the interaction itself and the point of the interaction in the view's coordinate space.

These can be used to create specific user activities for different regions of the view.

The closure should return an activation configuration with the user activity.

The interaction also takes an error handler.

While the interaction is disabled on platforms that don't support multiple scenes, errors can still occur due to configuration issues or a lack of system resources.

You've probably noticed that all of these ways of presenting window scenes use the same activation configuration object.

Its only requirement is a user activity, but it also contains scene request options and a targeted preview.

If these are not provided, the system will do its best to create them for you.

However, you can explicitly provide them to refine the experience.

As an example, these collection view cells have a thumbnail, title, and caption.

When the cell is pinched out, a scene is presented which mainly displays the larger version of the thumbnail's content.

Notice that the new scene is transitioning from the cell as a whole.

It would look better if it transitioned from just the thumbnail.

To achieve this effect, the activation configuration is created as before.

Then, check if the cell is a thumbnail cell.

If so, use the cell's thumbnail view to create a targeted preview and set it on the configuration.

By providing a custom preview on the activation configuration, the transition is much more polished.

Let's take a closer look.

Now, rather than transitioning from the cell itself, it transitions from the thumbnail, leaving the rest of the cell in place.

Activation configurations allow you to request any of your app's scenes.

However, you may simply want to display a file without having to build a scene yourself.

iPadOS 15 makes this very easy.

There's a special subclass of UIWindowScene .ActivationConfiguration called "QLPreviewScene ActivationConfiguration".

Returning a preview scene configuration requests a system-managed preview scene.

There's no scene delegate and no callbacks to worry about, but your app must declare support for multiple windows in the Info.plist.

Providing polished and convenient ways for people to open content in new scenes is important.

But it's equally as important to save and restore the scene's state so returning to them later on is a seamless experience.

When a scene moves to the background, the system will ask the scene's delegate for an NSUserActivity to represent its state.

This activity could be the root view controller's activity if it supports Handoff, or one created on the spot.

Here, a user activity is created saving the contents of a text field in the activity's user info dictionary.

To give the best experience, the scene state should go beyond just the content.

You should also save the visual and interaction state like scroll position, cursor position, and first responder status.

Rather than saving each of these independently, UITextField and UITextView now have an interactionState property.

This property provides a single object containing all of the interaction state.

The object doesn't contain the content itself, it's additional information for you to save in the user activity.

I've updated the previous example by saving the interaction state of the text field.

Now, by saving both the content and the interaction state, the user activity contains enough information to be accurately restored.

On iPadOS 14, restoring state can be a little tricky.

If you tried to restore state when the scene was connecting, you'd notice that the storyboard and views weren't fully loaded.

If you restored later, when the scene was transitioning to the foreground, you had to track whether or not it was for the first time.

iPadOS 15 solves this problem with a new delegate method explicitly for restoring state.

It's called after the scene is connected and the storyboard has been loaded, but before the first transition to foreground.

Regardless of whether your app uses storyboards, you should use this new callback for restoring state.

Restoring state from the activity saved in the previous example starts with the system calling sceneRestoreInteractionState.

Then, restoring the content of the text field.

Finally, restoring its interaction state.

It's critical that the content be restored before the interaction state.

Lastly, synchronously restoring state can be complex.

You may need to access a database or load a file and don't want empty UI during that time.

To account for this, iPadOS 15 allows your app to request a short-term extension.

During this extension, the launch image will remain visible while still allowing the main RunLoop to execute.

Once your content is loaded, the app should signal to complete the restoration.

This extension is short and not intended to be used for potentially long-running tasks like network access.

If your app fails to signal completion, or takes too long, it will be closed by the system.

To use extended state restoration, start by requesting an extension from the scene.

Then, kick off your asynchronous work.

Once the content is loaded, restore it and tell the scene to complete restoration.

The scene will then display your freshly restored UI.

Supporting multitasking in your app makes it shine, but to be a true next-level iPad app, there's more to do.

And now, over to Anant.

Anant Jain: Thank you, Chris! My name is Anant and I'm a UIKit engineer.

People love that iPad is light enough to hold in their hands while still being able to attach to a physical keyboard at a moment's notice.

They expect your app to keep up with them by having great keyboard support.

iPadOS 15 introduces several new features to bring your app's keyboard shortcuts to the next level.

iPadOS 15 features a totally new interface for discovering keyboard shortcuts.

It structures each command into familiar categories, bringing increased parity between the iPad and Mac Catalyst versions of your app.

The menu offers a convenient search feature that can be brought up from anywhere across the system.

You can even tap on a shortcut to trigger it.

If you need a refresher on how to support keyboard shortcuts with UIKeyCommand, including how commands are dispatched to the responder chain, check out "Support hardware keyboards in your app." On Mac Catalyst, apps each have a global menu called the main menu, which is displayed in the menu bar at the top of the screen.

The main menu consists of several category submenus, like File and Edit, and these category menus in turn contain further submenus with all the app's commands.

On iPadOS 15, we've brought the main menu system to iPad apps.

The new shortcut interface, which appears by holding down the Command key, displays this menu.

Now, the main menu system is represented a bit differently on the iPad compared to the Mac.

The Mac displays the full submenu hierarchy within each category, whereas the iPad flattens those hierarchies.

Disabled and non-performable commands appear grayed out on the Mac, whereas they are hidden on the iPad.

Furthermore, the iPad shortcut menu is designed to help discover keyboard shortcuts in an app.

It won't display keyless commands like the Mac menu bar does.

By default, the main menu contains all the system category menus like File and Edit.

These include all the system commands, like Undo and Redo, Paste and Match Style, and so on.

iPadOS 15 adds the Print command to this list of system commands, which apps can get by adding the UIApplication SupportsPrintCommand key to their Info.plist.

Now that iPad apps on iPadOS 15 support the main menu system, they can use the UIMenuBuilder API to customize it, just like Mac Catalyst apps.

In fact, if you already have a Mac Catalyst version of your app, you've already done most of the work.

Apps should use the builder to add all supported key commands to their main menus.

This is a change from how apps would previously declare keyboard shortcuts: by overriding the keyCommands property on UIResponder, or by calling addKeyCommand(_:) on a view controller.

Commands defined this way do still work, but they'll appear in a separate uncategorized section of the new interface.

Apps should remove key command declarations like this and instead add them to the main menu.

To customize the main menu, override buildMenu(withbuilder:) in your AppDelegate.

UIKit calls this method at app launch and passes in a UIMenuBuilder object.

Apps should check if the builder is modifying the main menu system.

If so, they can use the builder to make customizations.

Suppose an app wants some key commands in the File menu for working with tabs.

The app simply creates a submenu using the UIMenu API and adds the desired key commands as that menu's children.

Then the app calls the insertChild method on the builder object to insert the submenu into the File menu.

To reference an existing element in the main menu system -- like the File menu, in this case -- apps should specify the element's identifier.

The built-in system menu identifiers are defined as constants under UIMenu.Identifier.

Apps can easily create their own menu categories, too.

Here, the app creates a Bookmarks menu -- once again, using the UIMenu API.

Then, the app uses the builder to insert that menu into the root menu -- in this case, after the system View menu.

That's all it takes! Now, the builder can be used to make further insertions into the Bookmarks menu, just as it would with the File menu.

Just pass in the new menu's identifier, which UIKit automatically generates.

Now, I keep mentioning identifiers.

UIMenuBuilder will enforce that each element in the main menu system has a unique identifier, including individual commands.

Suppose an app inserts key commands to display content either as a list or as a grid.

Both commands share the same action: changeViewMode(_:).

In the main menu system, commands are implicitly identified by their actions, so both of these commands share the same identifier.

UIMenuBuilder won't allow both commands to be inserted unless they have different identifiers.

One way to distinguish these commands is to give them different propertyList values.

But a better way is to simply give each command a unique action describing what it specifically does.

The builder also enforces that keyboard shortcut combinations in the main menu are unique.

Suppose an app inserts a Get Info key command on Command-I.

Because the system Italic shortcut in the text style menu shares the same shortcut, this insertion will also fail.

Once again, there are two solutions.

The app can change the Get Info shortcut to something that doesn't collide with an existing shortcut, such as Control-Command-I.

Alternatively, the app can tell the builder to remove the text style commands if they're not needed.

If an insertion includes a duplicate, then UIMenuBuilder fails that insertion and logs an error in the console showing either the duplicated key commands or the shared identifiers.

If you're finding your insertions aren't appearing, chances are there's a duplicate somewhere, so look for logs like this in the console.

After the call to buildMenu(with builder:) finishes, the app's main menu appears in the Mac menu bar and the iPad shortcut overlay.

There is one problem, though.

The app added a submenu with commands to sort bookmarks either by name or by date.

But because the iPad shortcut overlay doesn't display the submenu hierarchy, it's not clear what these shortcuts do on iPadOS.

For situations like this, set a more descriptive discoverability title on the key commands.

iPadOS prefers the discoverability title over the regular title if both are provided.

Now, I mentioned earlier that individual responders should avoid declaring keyboard shortcuts and instead declare them in the main menu system.

However, responders should still implement action methods for main menu commands.

When key commands are triggered, UIKit automatically dispatches the action to a responder.

UIKit does this by traversing the app's responder chain.

As soon as it finds a responder that can perform the action, it calls that responder's action method.

If nothing in the chain can perform the action, then the key command isn't performable.

If you're new to UIResponder concepts, "Support hardware keyboards in your app" has a fantastic introduction to how it works.

"Qualities of a great Mac Catalyst app" also dives into a bit more detail.

When UIKit performs a responder search along the chain, it calls two useful UIResponder methods.

Apps can override these methods in their responders to improve their key commands.

The first is canPerformAction (_:withSender:), which UIKit uses to check if a responder can perform an action.

By default, this returns true if the responder implements that action method; otherwise, it returns false.

Responders can override this to add custom logic.

For example, a web browser can tell UIKit that the closeTab command isn't performable if there are no open tabs.

Since UIKit can't find a target responder for that action, the command will be non-performable and it won't be shown in the shortcut interface.

Note that overrides for this method must call super for unhandled cases.

The other useful method is validate(_ command:).

When UIKit finds a target responder for a key command, it calls this method on that responder and passes in a copy of the command.

Responders can override validate(_ command:) to update the appearance of the command for the current state of the app.

Here, the app updates the title of the toggleBookmark(_:) command based on whether or not the current page is bookmarked.

The title set in this method will reflect in the shortcut interface when it appears.

In iPadOS 15, UIKit is introducing a major change to the responder chain.

When apps adopt keyboard navigation with the focus system, then responder traversal will begin at the focused item rather than the first responder.

This change plays incredibly well with key commands.

Photos, for example, allows users to work with their photo library using just the keyboard.

When users focus a cell in the grid, they can hit the Space bar to view that photo.

They can even hit Command-C to copy that photo for pasting into another app.

Each cell implements various key command actions, and since responder traversal starts at the focused item, then the key commands target the cell.

In short, the focus system levels up key commands and the responder chain.

Take advantage of it to support powerful contextual shortcuts in your app.

To learn more, check out "Focus on iPad keyboard navigation." Finally, iPadOS 15 and macOS 12 introduce keyboard shortcut localization.

When you build your app with these SDKs, the system automatically localizes shortcut modifiers and inputs for each keyboard layout.

Take the Command-backslash shortcut, for example.

While the shortcut works on a US keyboard, it's impossible to perform on a Japanese keyboard, since that layout has no backslash key.

So the system will remap the shortcut for Japanese keyboards.

This means apps should not localize shortcut modifiers or inputs, and instead let the system do the work.

Apps can also opt out of automatic localization, either at the app-wide level or on a per-shortcut basis.

When the system localizes shortcuts, it also mirrors them for right-to-left layouts.

For example, a shortcut to navigate backwards with Command-left bracket is flipped to Command-right bracket.

If a shortcut shouldn't be mirrored, set the command's allowsAutomaticMirroring property to false to disable mirroring without disabling automatic localization entirely.

iPadOS 15 is a huge release for keyboard shortcuts.

There's a ton of other enhancements on top of everything we've covered today.

So you've adopted all the new Multitasking features, and you've built great keyboard support.

Now, Mohammed will give you a few pointers to take your iPad app to the highest level.

Mohammed Jisrawi: Thanks, Anant! Hi, I'm Mohammed.

Let's talk about the improvements iPadOS 15 brings to the system pointer.

iPadOS 13.4 introduced an adaptive system pointer that bridges between the touch-based UI of the iPad and the precision of a mouse or trackpad.

If you're new to pointer interactions, take a moment to familiarize yourself with them and check out these previous videos.

"Build for the iPadOS pointer" breaks down the pointer interaction APIs.

"Design for the iPadOS Pointer" dives into the design philosophy behind them and discusses best practices when adopting them in your app.

iPadOS 15 brings over some familiar Mac user interactions in a way that's consistent with the iPad's design language.

It also introduces some new concepts that enhance usability and clarity.

The first of these additions is band selection, a new pointer-specific multiselection experience that should be familiar to anyone who's used a Mac.

In iPadOS 15, when you click and drag in a collection view, the pointer stretches into a rectangle, and the collection view selects the items that the rectangle encompasses.

Of course this translates naturally to the familiar Mac UI on Mac Catalyst apps.

This interaction is built into non-list UICollectionViews.

Any UICollectionView that supports the existing one- and two-finger multiselection gestures via the shouldBeginMultiple SelectionInteraction API gets this behavior automatically in iPadOS 15.

For anything other than a UICollectionView, the new UIBandSelectionInteraction API allows you to easily adopt this experience in your app.

Since the selection logic is completely up to you, you're able to support custom selection behaviors and have your UI react to the changing selection in any way you choose.

To get started, instantiate the interaction with a selection handler to be called as the pointer moves and the interaction's state changes.

Once it's created, add the interaction to your view as you would any other UIInteraction.

In the handler, you can implement your custom selection logic by responding to changes in the interaction's state and selectionRect.

Here we have a simple implementation that sets the selection to the items within the interaction's selectionRect while the pointer is moving.

It then finalizes the selection session when the primary mouse button is released and the interaction ends.

In addition to basic selection, UICollectionView's built-in band selection supports some common keyboard shortcuts out of the box.

Holding Shift at the start of the drag, for example, causes items to be added to an existing selection rather than replacing the currently selected items.

Holding Command toggles the selection state of items in the selection rect.

This can be implemented in custom UI using the interaction's initialModifierFlags property, which provides the keys held at the start of the drag.

Since it's a set of all the pressed modifiers, you can respond to arbitrary key combinations to support custom or more advanced behavior that's specific to your app.

The second addition to the system pointer is the ability to attach accessories.

Accessories communicate additional information and provide contextual hints by combining secondary shapes with the primary pointer.

For instance, in the example on the left, the two arrows indicate that this view can be dragged horizontally.

In the example on the right, the plus provides some additional context for this Add to Cart button.

There are a few key distinctions between accessories and using a custom pointer shape.

Accessories are visually separate and secondary to the main pointer.

This is emphasized by the fact that they may be rendered with a different appearance and animate separately from the main pointer.

They are independent units which can be combined and positioned around the pointer to communicate different ideas.

Since they are independent, they can be combined with any pointer style.

Here's a demonstration of how the same set of accessories -- two arrows indicating draggability -- can be combined with different pointer styles.

On the left, they're combined with a lift effect where the pointer merges with the view and lifts it up.

In the middle, they're alongside a highlight effect, where the pointer turns into a rounded rectangle and goes under the view.

On the right, they appear with the default system pointer using the new UIPointerStyle.system() API.

So we're able to provide this additional context while still using the ideal effect for the situation, without sacrificing the liveliness of the pointer, and while still maintaining its deep relationship with the app's UI.

Just like it animates between pointer styles, the system will automatically animate the appearance and disappearance of accessories.

It'll also seamlessly animate between accessory shapes and position.

The act of transitioning accessories while a given effect is active can be meaningful.

Such a transition can be used to communicate a change in state or behavior in the underlying UI.

In the example on the left, the transition from a plus to a no sign may indicate that an operation that was previously possible is no longer permitted.

Pointer accessories are composed of a UIPointerShape and a UIPointerAccessory.Position, which describes the desired position as an offset from the pointer's midpoint and an angle from the top.

For convenience, UIKit provides some predefined values for positions around the pointer.

If the predefined positions don't quite fit your needs, you can use them as a starting point and customize individual properties.

This example starts with the topRight position and customizes the offset.

You can also define entirely custom positions, as in this example, which creates a position with a custom offset and angle.

To set up the example we saw at the beginning of this segment, we'll create a UITargetedPreview with our view and use that to create a UIPointerStyle with a lift effect.

Then we'll set the style's new accessories property to an array that contains our two arrow accessories.

UIKit provides a premade arrow accessory, so we can just create two of those positioned on the left and right sides of the effect.

So now when the pointer hovers over this view, the two arrows animate out as the view lifts, hinting that it can be dragged.

If you've ever tried to implement this sort of interaction where a view with a pointer effect is draggable, you've probably noticed something like this.

When the pointer reaches the edge of the pointer region, it disengages from the lifted view and the effect ends.

This is usually desirable, since it prevents the pointer from sticking to views as it moves around.

However, in scenarios like this, the ideal experience would be for the pointer effect to remain stable and latch to the view and follow it along as it's dragged.

To better enable these sorts of interactions, iPadOS 15 introduces the concept of latchingAxes on UIPointerRegion.

When a region latches along a given axis, its pointer effect follows the pointer along the axis when the primary mouse button is pressed.

A horizontally latching region lets you drag freely along the x-axis while still rubberbanding along the y-axis.

A vertically latching one lets you drag freely along the y-axis.

And one that latches along both lets you drag freely along both axes.

These new tools can be used to build some really powerful new experiences.

Here they are combined in a document editing app like Pages or Keynote.

This image can be selected using band selection.

Selecting the image causes dragging indicators to appear, and when the pointer hovers over an indicator, accessories appear to hint at how the image will be resized if it's dragged.

And finally, latching allows the pointer effect and accessories to follow the axis-locked resize gesture.

These are just some of the enhancements iPadOS 15 brings to the iPad.

Take advantage of them where applicable to really maximize the utility of your app.

Adopt prominent scenes to give people a focused, uninterrupted view of your app's content.

Empower them to achieve complex tasks quickly with the new keyboard shortcuts menu, and boost their productivity with the new pointer features.

Thanks for watching! ♪