Handle trackpad and mouse input

Hello and welcome to WWDC.

I'm Steve Moseley, a UIKit engineer, and this is "Handle Trackpad and Mouse Input." In this video, we're going to talk about ways to make your app feel responsive to the indirect input mechanisms like trackpads and mice introduced in macOS Catalina and iPadOS 13.4. There are some common updates that apply to every app, and some advanced updates for apps that want to go the extra mile.

In common updates, we'll go over handling pointer movement, locking the pointer, and handling scroll input and trackpad gestures.

In advanced updates, we'll go over handling button mask and keyboard modifiers, accepting or rejecting events with new UIGestureRecognizer and UIGestureRecognizerDelegate methods, distinguishing touches from indirect input devices, and opting in to some new behavior with an Info.plist key.

Let's take a look at those common updates.

With a mouse or trackpad, users expect to interact with your application without touching the screen. Notice how Safari reveals their toolbar when the pointer moves into that region, or how they show the tab-close button when the pointer moves into a tab.

Safari responds to pointer movement like this with UIHoverGestureRecognizer.

UIHoverGestureRecognizer was introduced with Mac Catalyst in Catalina and is now available in iPadOS. It is a normal gesture recognizer and works on iPad just like it does on the Mac. If you're overriding UIApplication.sendEvent, you'll notice it's driven by a new event type, EventType.hover.

You instantiate a UIHoverGestureRecognizer with a target and action, just like you would any other gesture recognizer. In your action callback, you'll switch over the gesture state and perform appropriate actions.

Note that gesture state "began" maps to the pointer entering your gesture view's bounds...

and "ended" maps to the pointer exiting your gesture view's bounds. Here, we're revealing video playback controls based on whether the pointer is within our view.

If you inspect touches in your iPad or Mac Catalyst apps, you'll see there are a few new phases to track pointer movement.

These phases map to the overall pointer movement within your window. RegionEntered means the pointer has entered your window. RegionMoved means the pointer is within your window but has not clicked or pressed down yet. RegionExited means it has left your window.

Notice that these phases do not always align with the gesture states we discussed for UIHoverGestureRecognizer.

The UIHoverGestureRecognizer states will only map to what the pointer is doing within the bounds of your gesture's view, whereas these phases pertain to what the pointer is doing within your window. Use UIHoverGestureRecognizer to respond to pointer movement or for hiding and revealing content, like we saw in the Safari example earlier.

Don't use it to modify the appearance of the pointer or apply a hover effect. For either of those, you should use UIPointerInteraction.

For more on UIHoverGestureRecognizer, see our talk from 2019, "Taking iPad Apps for the Mac to the Next Level." And if you are interested in modifying the appearance of the pointer, check out "Build for the iPadOS Pointer." In addition to responding to pointer movement, some apps, like games, would like to lock the movement of the pointer. New in iPadOS 14 and Mac Catalyst Big Sur, we've introduced API to allow you to do just that. It's really easy to use.

Set your lock preference with UIViewController API and observe the resolved value with the new UIPointerLockState. That's it. The pointer is a shared resource, so ultimately, the system decides whether the pointer should be locked. That means your preferred pointer lock state may or may not be honored. Let's see how these two pieces of API work together.

Your view controller sets a prefersPointerLocked value of "true." As long as some requirements are met, the system sets the lock value of your scene to "true" as well. That status is reflected in the UIPointerLockState of your scene.

What happens if you need to present content and you want to disable the lock? Let's say your game experiences a networking error, and you'd like to present a UIAlertController. The user expects to interact with that content using the pointer just like they can throughout the rest of the system.

No problem. The default value of prefersPointerLocked is "false." When UIAlertController is presented, its value of prefersPointerLocked is observed by the system and the pointer lock is disabled.

As view controllers are presented or dismissed, your scene's pointer lock value is automatically updated, which means you don't need to keep track of this state.

If you want the pointer to be locked in your scene, just override prefersPointerLocked property of your view controller to be "true." If at some point you want to disable or change this value, just call setNeedsUpdate OfPrefersPointerLocked.

If there's a part of your application that needs to see the current lock status, you obtain the pointer lock state from your scene and look at the isLocked property.

Here, we have an object that wants to be notified of changes to the pointer lock state of its scene.

It obtains the pointerLockState object, then registers to observe that object.

When UIPointerLockState .didChangeNotification is posted, the closure will be executed and the isLocked value of UIPointerLockState will be passed on to another part of the application.

As mentioned in the diagram, there are some requirements your scene needs to fulfill in order for your preferred pointer lock value to be considered. The requirements are different per platform, so let's start with iPadOS.

First, your scene must be full screen. This means your application cannot be in Split View multitasking or Slide Over. It also means there can be no other app in Slide Over.

Full screen here does not mean using the UIRequiresFullScreen Info.plist key, simply that your scene must occupy the entire screen.

Second, your scene must be in the foregroundActive activation state. This means it cannot be deactivated for any reason, like Control Center or Notification Center being presented.

In Mac Catalyst, your application must be the frontmost application for your prefersPointerLocked value to be considered by the system.

And if you have multiple windows, the window where you'd like the pointer to be locked should also be ordered to the front.

If your application fails to meet these requirements, the pointer lock is disabled.

On iPadOS, if a slide-over app is displayed, or on macOS, if your application is no longer frontmost, isLocked will change to "false," and you will be notified through UIPointerLockState DidChangeNotification.

However, you don't need to do anything to get the pointer lock back. The system continuously evaluates these requirements. So, as conditions change, so will the pointer lock status of your scene. You don't need to call setNeedsUpdate OfPrefersPointerLocked.

Remember that it's the system's discretion to lock the pointer. These requirements are subject to change and are informed by user behavior. So your application should not assume that your prefersPointerLocked value will always be honored. You should always observe changes to isLocked and respond in your application appropriately.

Finally, pointer locking is not available on all scenes. For those, the pointerLockState property on UIScene will return "nil" to indicate that locking is not available. When locking the pointer, you'll also want to look at relative movement from trackpads and mice.

For more information on that, see the "Bring Keyboard and Mouse Gaming to iPad" video.

Let's talk about handling scroll input.

It is important to ensure that all regions of your app respond correctly with connected pointing devices. If users can pan something with their finger, they'll expect to pan the content with either a two-finger gesture or a mouse scroll wheel.

Here we see that the custom controls in Control Center have been updated to pan with two-finger swipes on a trackpad.

You handle scroll input in your app by updating UIPanGestureRecognizer's allowedScrollTypesMask property. Simply give it the set of scroll types you want to handle and that will enable EventType.scroll support for your gesture.

UIScrollView's pan gesture recognizer updates allowedScrollTypesMask to handle all types of scroll input, but standard UIPanGestureRecognizers have no mask by default. So you'll want to update this property for all your application's pan gestures.

Let's say you have an app that hides content on either side of its main view. The user can reveal this content with a horizontal swipe, which is powered by a pan gesture. Your designer has determined revealing this content with a scroll wheel doesn't feel very natural, so you're going only support continuous scroll types with this gesture.

Simply update the pan gesture's allowedScrollTypesMask to UIScrollTypeMaskContinuous. Perhaps there's a custom pull-to-refresh interaction in your application that's also powered by a pan gesture. For this gesture, you might want to respond to all types of scroll input, so just update its allowedScrollTypesMask property to UIScrollTypeMaskAll.

Handling pinch and rotate trackpad gestures is even easier. Just use UIPinchGestureRecognizer and UIRotationGestureRecognizer. To ensure every app can handle these types of indirect gestures, these recognizers use a compatibility mode. By default, they are driven by gesture-simulating touches. UIKit creates these touches a fixed distance apart and simulates their movement in response to movement on the touch surface.

Starting in iPadOS 13.4 and macOS Catalina 10.15.4, applications can move these gestures out of the compatibility mode and respond to a new event type, EventType.transform.

This event type comes directly from the input device which enables precise pinch and rotate gestures like your users expect.

To get this new event type and to move these gestures out of their compatibility mode, you'll need to add a key to your application's Info.plist. We'll talk more about that in a bit.

The good news is that there is no additional code required for either of these scenarios. With or without the key, UIPinchGestureRecognizer and UIRotationGestureRecognizer know how to handle this input. And when you do add the Info.plist key to your app, you don't need to write code to handle the new event type. It just works.

Do note, though, that if you're adopting the Info.plist key, these gestures will no longer be driven by touches during trackpad input. In that case, numberOfTouches will return zero and locationOfTouch:inView may throw an exception.

And those are the common updates that apply to every app.

Let's move on and talk about some advanced updates to surprise and delight your pro users. Button mask and key modifiers are a great way to add advanced functionality to your application.

Context menus use button mask to recognize two-finger taps and secondary clicks so they can provide a more streamlined UI.

And Numbers uses key modifiers so I can select multiple rows with the pointer and Shift key modifier, just like I can with my finger.

UIEvent.ButtonMask is a new type in iOS, and it's the set of buttons pressed while clicking with a pointing device.

It's present on both UIEvent and UIGestureRecognizer as buttonMask, giving you a convenient way to respond to just the primary button of the device, create features that respond to two-finger taps and secondary mouse buttons, or target high-number mouse buttons. Note that the button mask on UIGestureRecognizer is from the last event processed.

If you want a simple way to require a specific button mask before firing, we've updated UITapGestureRecognizer with buttonMaskRequired. Just give it a button mask, and you're done.

There's even a convenience function on ButtonMask that returns the appropriate mask value for high-number buttons. Together with buttonMaskRequired, it's really easy to target high-number mouse buttons as accelerators for advanced functionality in your app.

If you've used UICommand or UIPointerInteraction, you're familiar with UIKeyModifierFlags. It's the set of keyboard modifiers pressed during an event. We've brought UIKeyModifierFlags to both UIEvent and UIGestureRecognizer as modifierFlags. This property can be used during gesture callbacks to alter how you respond to an event.

For instance, clicking on a link in Safari with the Command key pressed causes the link to be opened in a new tab. Like buttonMask, UIGestureRecognizer's modifierFlags is populated from the last event processed.

For more on how to have a great keyboard experience in your app, check out the "Support Hardware Keyboards in Your App" video.

Button mask and key modifiers are easy to use.

Targeting a third mouse button is as easy as using buttonMask.button to get the appropriate mask and setting the result on UITapGestureRecognizer. buttonMaskRequired. That's it.

Let's go back to our UIHoverGesture Recognizer example from earlier to see how modifier flags work. Previously, we revealed video playback controls whenever the pointer entered our gesture's view.

If we optionally want to show chapter-selection controls whenever the UIKeyModifierAlternate is pressed, we just need to check if modifierFlags contains that value. Button mask and modifier flags are especially powerful when combined with new API for UIGestureRecognizerDelegate and UIGestureRecognizer subclasses. These methods are called for only the events handled by your gesture, so UIPinchGestureRecognizer won't be asked about EventType.scroll. These methods give you an opportunity to accept or reject those events based on button mask, modifier flags, or other properties. Note that these methods happen before the event is fully processed by the gesture, so UIGestureRecognizer's buttonMask and modifierFlags properties will not include the new values found in the event.

If you're inspecting either of those properties in these methods, you should look at the values on UIEvent, not the ones on UIGestureRecognizer. As gestures like UIPanGestureRecognizer and UIPinchGestureRecognizer respond to multiple non-touch-based events, you should move any event-related code in methods like gestureRecognizer(shouldReceive touch:) into either of these two new methods.

Let's look at some examples for how you can use this in your app. We have a UIGestureRecognizer subclass that only wants to receive events with a buttonMask of secondary. You might do this for functionality driven exclusively by two-finger taps or secondary mouse-button clicks.

Start by overriding the gesture subclass method shouldReceive(_ event). In that method, you simply need to check if the buttonMask on the event is exactly equal to secondary. If it is, we receive the event. If not, we reject it.

As mentioned earlier, while buttonMask exists on UIGestureRecognizer as well, we shouldn't look at that property in this method. ShouldReceive(_ event) happens prior to the event being fully processed by the gesture, so UIGestureRecognizer's buttonMask will not be up to date at this point.

It's common to allow click plus the Control key modifier to perform the same actions as a secondary click. We can update our example for that as well. We just need to modify our shouldReceive method.

First, check if the buttonMask is exactly primary.

If that's true, we check if modifierFlags is equal to UIKeyModifierControl.

We'll receive the event if it's a secondary click or Control click and reject it if it isn't.

Let's bring back our video example again. We want to add another hover gesture over the video that shows closed-caption controls. The user can already get to this feature through a settings menu, but we'd like to give them a quick way to do this with a key modifier and the hover gesture.

We can instantiate our HoverGestureRecognizer like before.

This time, we'll set ourselves as the delegate, and implement the gestureRecognizer shouldReceive event method.

In that method, we'll receive the event if UIKeyModifierAlternate is pressed, and reject the event if it isn't.

As you're considering how to polish pointer support for your app, you may want to distinguish touches originating from a pointing device from ones originating from a finger. You especially may want to consider this if you have a lot custom hit-testing code in your app.

As the pointer is more precise than a finger, you can reduce any expanded hit-testing regions for those touches, providing a more precise experience. Touches from a pointing device are given the new TouchType of indirectPointer if you opt in to the UIApplication SupportsIndirectInputEvents Info.plist key.

You can use this touch type with existing API like UIGestureRecognizer. allowedTouchTypes to have gestures that only respond to pointer clicks, or ones that only respond to finger-based touches.

Let's talk some more about UIApplication SupportsIndirectInputEvents. It's a Boolean key you add to your application's Info.plist. This key is not required to enable pointer interactions, button clicking, scroll input, or trackpad gestures. All of those work with or without the key. It is required in order to get the new touch type indirect pointer and EventType.transform.

Existing projects do not have this key set and will need to add it. Starting with iOS 14 and macOS Big Sur SDKs, new UIKit and SwiftUI projects will have this value set to "true." In a future release, the default will change and we will no longer consult the value of this key. Let's see exactly what happens if this key is or is not present.

It's helpful to think of UIApplication SupportsIndirectInputEvents as opting out of a compatibility mode.

We added this compatibility mode so that users would have a great initial experience with indirect input on iPadOS 13.4.

So if the key is not present, as it is for all existing projects, your application is in this compatibility mode.

Clicks from pointing devices are TouchType.direct, the same as for finger-based touches, so you won't be able to distinguish them apart. Pinching and rotating on the trackpad result in gesture-simulating touches that may incidentally activate other gestures.

If the key is present and true, your application is out of the compatibility mode and new features are enabled. Clicks from pointing devices are TouchType.indirectPointer, allowing you to target and modify functionality for precision pointing devices.

Pinching and rotating on the trackpad emits a new event type directly from the input device: EventType.transform. This enables precise pinch and rotate gestures that won't incidentally activate other recognizers.

With this key, you are fully entering the new world of indirect input on iPadOS and Mac Catalyst. With that, there are a few things to be aware of.

New event types like EventType.scroll or EventType.transform are not touch-based, so you'll need to be careful with touch-related gestureRecognizer API.

When UIPanGestureRecognizer UIPinchGestureRecognizer or UIRotationGestureRecognizer are driven by these new event types, numberOfTouches will return zero and locationOfTouch:inView may throw an exception.

Also be aware that any code you may have in your shouldReceivetouch delegate methods for these gestures will not be run when they're driven by these events. After opting in to this key, UIPinchGestureRecognizer and UIRotationGestureRecognizer are removed from their compatibility mode. So any incidentally activated gestures from that mode will no longer be triggered. In this new world of indirect input, gestures respond to multiple types of events. Because of that, you may find it helpful to detect which event your gesture recognizer is responding to. You can use the shouldReceive methods we discussed earlier to help with that. When responding to EventType.touches, you can use API like numberOfTouches or locationOfTouch:inView. If you're responding to other events, you should avoid those methods.

There are some simple things you can do to make your app come alive with trackpad and mouse input.

Enable scroll input for your pan gestures. Respond to pointer movement by hiding or revealing content.

Add the Info.plist key to your app to gain the new TouchType and EventType, allowing you to customize functionality for pointer-based touches and have precise pinch and rotate gestures in your app.

Use new event properties and gesture recognizer API to delight your users with alternate responses to button presses and keyboard modifiers. Thanks for watching this video. I can't wait to try out your updated apps.