Build for the iPadOS pointer

Hello and welcome to WWDC.

Hi. I'm Mohammed, a UIKit engineer. Later, I'll be joined by my colleague Joey from the iOS System UI team, and this is "How to Build for the iPadOS Pointer." In 13.4, iPadOS added general pointing device support to the iPad. This is a brand-new input method for an OS that has been primarily touch-based since its inception. Rather than just bring over the exact interaction model that exists on a Mac, we thought about what value a pointer can add to iPadOS and how it can coexist with a touch-based interface. The result is a system built on a fluid adaptive pointer that morphs into controls when hovering over them to adapt its accuracy and provide a clear indication that you're about to interact with them. In other contexts, the pointer changes its shape and motion characteristics to provide contextual hints. When hovering over text, for example, the pointer changes into a beam and snaps to lines to make interacting with text easier. In this video, we'll discuss the strategy we followed to update iPadOS for pointer and how you can use the same strategy when updating your app. In the course of the discussion, we'll talk about the pointer customization APIs introduced in iOS 13.4, and we'll talk about some of the best practices to keep in mind while updating your app. We'll also touch on some of the basic design principles behind the pointer UI. For a more detailed discussion of these principles and some of the thought processes behind the pointer design, check out the session on "Design for the iPadOS Pointer." When you use an iPad with a pointing device connected, you'll notice that a lot of things work with the pointer automatically without any additional adoption from apps. This is because a lot of system components have pointer support built in. Controls like UIBarButtonItem, UISegmentedControl and UIMenuController, to name a few, have built-in pointer effects and behaviors. Scroll views respond to scrolling with two fingers as well as mouse wheels and pinching to zoom on the trackpad. And in addition to these scroll view basics, collection and table view now support two-finger panning for swipe actions. UITextView and other components that use UITextInteraction support a suite of new quick text selection and editing gestures that should be familiar to anyone who's used a Mac. UIDragInteraction now allows you to drag quickly via a click-and-drag instead of requiring a long press as it does with touch. And UIContextMenuInteraction lets you invoke its menu in a new, compact appearance via a secondary click. So, how do we go about updating an app for pointer? We found that it's best to take a top-down approach. Start with the higher-level APIs. They'll offer you the most polished experience with tuned behaviors and visuals that are consistent with similar UI in the rest of the system. There are a number of pointer-related APIs available to you at different levels of the stack. Many controls have built-in pointer effects. Some, like bar buttons and segmented controls, have their effects enabled by default, while others, like UIButton, offer API that allows you to enable and customize their effects. With UIPointerInteraction, you can make your custom UI react to and interact with the pointer in a way that feels consistent with the rest of the system. Using the interaction, you can choose from one of a collection of system-vended effects to apply to your views. Or you can change the pointer's shape within an area of your app. UIHoverGestureRecognizer lets you respond to the pointer's motion more directly. This is great for any custom behavior that doesn't involve applying hover or highlight effects or modifying the pointer's appearance. For more details on this gesture and other pointer-related additions to gestures and events, check out the session on handling trackpad and mouse input. When updating your app, you want to aim for an experience that makes it feel consistent with the rest of the OS. When deciding where to add pointer support, take your cues from the Human Interface Guidelines and built-in apps. To that end, a good starting point is ensuring that controls in your app's chrome have the appropriate effects. By "chrome," we mean the application's top and bottom bars, so start with any bar buttons in the app. UIBarButtonItems created using the system item, image or title APIs get appropriate pointer effects automatically. If you're using the custom view API, you'll have to implement the pointer behavior yourself, either using the view's convenience API, if it has one, or by installing a pointer interaction and managing the effect yourself.

UIButtons that have been placed in bars via the custom view API have their built-in interactions enabled by default and are given an effect that the system has deemed appropriate for that button in its containing bar. You'll be able to tweak this effect using UIButton's convenience APIs. To make this really easy, UIButton has a two-stage convenience API. To enable the automatic effect, simply set the button's isPointer- InteractionEnabled property to "true." After enabling the effect, you may customize it using the button's pointerStyleProvider. In this closure, the system offers you a proposed effect and shape that have been determined based on the appearance, size and contents of the button.

Here, you can customize either of these or replace them entirely and construct a new style. Before we look at some examples, let's take a brief detour and talk about pointer styles. All APIs that modify the appearance of the pointer use UIPointerStyle to describe the modifications they apply. Styles fit into two categories.

The first is what we call a content effect. Content effects usually cause the pointer to morph into a view in the app and apply some visual treatment to it. A common example of this is the highlight effect applied to bar buttons where the pointer morphs into a rounded rectangle, slides under the button and applies a subtle parallax effect to it.

This style consists of a UIPointerEffect which describes the visual treatment applied to the view... and a UIPointerShape which describes the shape to which the pointer will change.

To specify this effect, we construct a UIPointerStyle with the highlight effect and roundedRect as the pointer shape.

The second category is shape customizations, which are expressed via a UIPointerShape and a UIAxis mask.

When applied, the pointer morphs into the shape and is constrained along the specified axes within the current region. A good example of this is the pointer behavior in text views where we use a verticalBeam as the shape and specify "vertical" as the constrainedAxes to make the pointer feel as if it's on a horizontal rail along the line of text. So, with that basic overview of the control APIs and the pointer styles, I'm going to hand it over to Joey to demo how they can be applied to an existing app. Joey? Thanks, Mohammed. Today we're going to take a look at an app that I've been working on called Quilt Simulator. I'm not much of a quilter myself, but I've been learning about quilting as I built this app. Let me build and run to see what we've got so far.

The first thing we want to take a look at is this button in the lower right corner. It toggles a straight-line guide mode on and off to make stitching in a straight line easier. You can see that it uses the highlight effect, but it doesn't seem right. The effect isn't the right size. Let's switch back to Xcode and see what we can do to improve it.

Here we can see that the pointer interaction has already been enabled for the button which gives up the default effect. We can add a pointerStyleProvider for this ruler button to improve the shape.

First we create a rect that is outset from the button's bounds by a suitable amount...

then convert that rect to the coordinate space of the target of the preview of the proposed effect. And finally, return a style that uses the proposed effect but with an improved pointer shape based on that rect. Let's take a look at the change.

Much better. This button looks right and feels more comfortable. Next, let's turn our attention to the thread selector button in the upper right corner. This button changes color with the currently selected thread color.

I already customized this bar button to use the lift effect because it has an intrinsic shape. You can see here it doesn't look quite right. When the pointer approaches the button, we see a diffuse blur under the button. This is distracting and incorrect. Let's switch back to Xcode to address this problem.

Since it is a UIBarButtonItem, it already gets an effect by default. Again, we modify the pointerStyleProvider, this time to change the default effect to a lift effect. This implementation isn't correct though, so let's fix it with the correct shape.

In this updated implementation, we create a new UITargetedPreview that's suitable to be used with the lift effect. It's created with the view, target as well as preview parameters which we create above.

The parameter's shadowPath property is set to be the path which matches the outline of the button.

Finally, the closure returns a pointer style using a lift effect created with this preview and a pointer shape that matches the button. And now back to the iPad to see the improvement.

Now things are looking good. That spool of thread just comes alive as we approach it with the pointer. Mohammed, back to you. Thanks, Joey. Now that we've seen how to enhance controls, let's talk about custom UI. When considering adding pointer behaviors to custom UI, focus on areas where you think they'll add utility and unique value. Our Quilt Simulator app has this large area in the middle. You click on it once to begin a stitch, and then you click again in a different location to complete it. By implementing some custom pointer behaviors, we can do some things to improve the experience. Let's add two enhancements. First, let's change the pointer's shape when hovering over the quilt area to make it clear that clicking in a location starts a stitch.

And let's add a guided mode for beginner quilters where the pointer is constrained along the vertical axis to make it easier to make perfectly horizontal stitches.

Since this is an entirely custom view, we'll use UIPointerInteraction directly to achieve our goals. As with other UI interactions, we create an instance of UIPointerInteraction and attach it to the view. Unlike others, however, this interaction's delegate is optional.

As we did with the controls in our app, we'll be using UIPointerStyles to specify the effects or shape changes we'd like to apply to the pointer. Since we want fine-grained control over where the styles are active, we'll also be defining custom UIPointerRegions to indicate to the interaction where to apply the styles we provide. By default, the interaction uses a region that covers the entire view.

If the delegate doesn't provide any styles, the interaction applies the automatic pointer effect to the entire view. This is really convenient in simple cases where we want to apply an effect to a view and be done with it. But our scenario's a bit more specialized. Since we want to define behavior within specific subregions of our view, we're going to implement UIPointerInteractionDelegate's regionFor request method. This method is called to request new regions as the pointer moves within the interaction's view. For our implementation, we'll provide it with regions that correspond to our guides. As soon as the pointer enters a region, the interaction calls its delegate's styleForRegion method to request a style, so we'll need to implement the method and return a shape customization style that provides this crosshair shape. Now I'm gonna hand it back to Joey to see what this looks like in practice.

Thanks again, Mohammed. Let's take another look at Quilt Simulator to see how we can improve it. Once the quilter has selected a thread color, it's time to start quilting, which is easily done by clicking for each of the stitches on the patchwork area. First I'll select some yellow thread.

I can zoom in with two fingers, leveraging the capabilities of UIScrollView...

and I'll make a few stitches here to get started.

This is a great opportunity to take advantage of the laser precision the pointer is capable of by specifying a custom pointer shape and assist the user with custom regions.

The first thing we want to do is add the custom pointer interaction to the view.

This creates a new UIPointerInteraction with "self" as the delegate and adds it to the quilt view.

I already implemented the styleForRegion delegate method. The first line creates a UIPointerShape object with a custom Bezier path we get from helper method.

The second creates and returns a UIPointerStyle that uses the pointer shape. Let's see what it looks like.

Much better. This feels more natural with such fine thread. Let's return to the straight-line guide I mentioned earlier. If I turn it on by clicking the ruler button...

then make a few stitches...

it's plain to see how much easier it is to produce excellent results. This feels a little disorienting though. The pointer is moving around freely, but the stitches are showing up only on the grid lines.

Let's see what we can do about that.

We can modify our delegate method implementations to restrict pointer movement vertically so the pointer is constrained to each of the rows. First, I'll add a regionFor request implementation to supply different regions per guideline.

If we're using StraightLineStitch mode, then provide a separate region that's the height of one of our grid lines.

Otherwise, return the default region. Next, we'll constrain the pointer's movement on the vertical axis.

If we're using the guide, return the crosshair with the constrainedAxes set to "vertical." Otherwise, the same as before.

As you can see, we've really enhanced the app for stitching straight lines. As I toss the pointer, it lands on horizontal guidelines, and the cursor is almost magnetically attracted to where it's supposed to be. And that's it for the demo. Back to you, Mohammed. Thanks again, Joey. That looks great. Now let's talk about a few extra steps you can take to tailor the pointer experience to your app.

There are a few little things you can do to really polish the pointer experience in your app. One such enhancement is providing expanded padding around views to amplify the pointer's magnetism and make it easier to interact with important elements in your UI. As you can see here, the pointer snaps to this button well before it reaches its edge. This is done by providing a pointer region that extends the pointer style's effective area. It's important to note that any regions you provide must be contained within a hit-testable area of the interaction's view. If you provide a region larger than the view, you must also ensure that this area hit-tests to the view by overriding its hit-test method. By tuning pointer regions, we can control exactly when and how the pointer transitions between effects. In this example from the Reminders app, we can see that even though these views are visually separate, the pointer transitions between them seamlessly without dropping back to the system shape in the gaps.

This was achieved by providing the interaction with contiguous pointer regions.

When coordinating multiple pointer regions or styles, it often makes sense to take your interaction up to a higher level in the view hierarchy where you have a more complete picture of the UI and can reason more globally. In this example, the interaction is attached to the entire view, and its delegate provides regions for the individual subviews.

Another great thing you can do is coordinate additional animations with the pointer interaction's animation. This is a great way to show useful information or hide extraneous chrome to reduce clutter during the effect. Here we see how UISegmentedControl hides its separator as the pointer shifts between segments for a cleaner, less cluttered look.

To coordinate animations, just implement UIPointerInteractionDelegates willEnter and willExitRegion methods and attach animations to the provided animator object. In this example, we'll fade out a separatorView when the pointer enters the region, and fade it back in when it exits. For more guidelines and polish tips, check out the Human Interface Guidelines and developer documentation. Also, please check out the sample related to this video as it has some great examples for you to try out. Take these simple guidelines to heart and build iPad apps with first-class pointer experiences. Prefer using higher-level APIs for a more native look and feel. Add pointer effects to controls and custom UI to make them feel responsive. And finally, take some small steps to polish your implementation and infuse your UI with surprise and delight. Thank you for watching. I can't wait to use all the amazing pointer-capable apps you're going to build. [chimes]