Sync files to the cloud with FileProvider on macOS

♪ ♪ Hi. I’m Johannes Fortmann from the Cloud File Providers team, and I’m here to show you file providers on macOS. If you’re a cloud storage vendor syncing your users’ files to macOS, you’re in the right talk. After the introduction, we’re going to talk about some of the user flows involved with syncing files. We’ll then run through one of the flows in Xcode and discuss the order in which you will implement support for each of them. We’ll have a quick overview of additional optional integration points and your next steps. First off, let’s talk about what file providers can do for you.

The File Provider framework allows you to integrate your cloud storage into the file system on macOS. It uses new APFS features to allow on-demand downloads of user files and folders. The API is entirely in user space. We’re deprecating kernel extensions on macOS, so this is a good alternative for you if you were relying on FUSE or KAUTH to intercept syscalls or download files on demand.

All you have to do is handle uploads, downloads, and tell us what changed remotely. The system will tell you what changed locally and handle all the rest. All of this functionality is well integrated with the system and in particular in Finder. Your provider will show up in the sidebar. File status will be shown and tracked in Finder, and there are several customizable integration points with the UI. You implement an app extension that integrates with the system. Its life cycle is driven by user actions. Initially, you create a domain which represents the file tree that the user can access in your cloud storage. The system will expose that domain in the Finder sidebar and create a root directory for the domain in the file system. At this point, no actual data is on the device, but the user can already start interacting with the root. How does this work? The root is what we call a dataless directory. It’s a new kind of object in APFS, and there are APIs to recognize them and interact with them. But more importantly, dataless objects are fully transparent to processes who happen upon them unprepared. Reads trigger downloads, and files lose their dataless property before the reads are allowed to resume. In this presentation, we will see how the File Provider framework allows you to implement callbacks that get called when processes read dataless files.

Let’s have a look at some user flows to get sync up and running. We’re going to look at four key flows that cover both sync down and sync up. In each flow, you’ll see that the system calls your extension whenever it needs new data.

We’ll show you that you can talk to your cloud server to pull that data and, finally, call a completion handler to reply. First, we’re going to look at what happens when a dataless file is read.

When the kernel detects a read access to a dataless file, that syscall is paused while your extension is called to fetch the contents of the file. Your extension’s fetchContents method is called. Typically, you will implement it to perform a download. When the download completes, it calls a completion handler. The contents of the file are handed to the system which fills in the formerly dataless file without invalidating the open file descriptor. The system then unpauses the read access. Now that the file is no longer dataless, subsequent reads won’t have to involve your extension. Enumerating a directory works very similarly. The kernel detects a readdir call and pauses it. It calls your extension to enumerate the items in that directory. You fetch the metadata for these items from your server. And you reply with a number of items. The enumeration is paginated. You can return less than the full set of items, and the system will pick up enumerating from where it left off. Once all pages have been enumerated, the system will allow the original call to go through. Like in the file case, once the directory has been enumerated, subsequent readdir calls will use the contents from disk and not have to involve your extension.

But what if those contents change remotely? Well, you will have to inform the system of the remote change. Let’s look at how that works. If there’s a remote change, your server can send a push notification to the Mac. In response to that push notification, you signal the system that there are changes that need to be enumerated from a special enumerator, the .workingSet. The system will turn around and enumerate the items that have changed in the .workingSet. A continuation token called the syncAnchor is used to enumerate only the new changes. This token is defined by your extension. The system keeps track of the syncAnchor it last enumerated to. It will call your enumerator with the enumerateChanges(from syncAnchor:) method. In response, you return changed items, and once you’re done, give us a new syncAnchor that we can use next time around. The system will asynchronously go and update the user-visible files. We use an APFS compare-and-swap feature which makes sure that local changes aren’t lost in the process. Furthermore, the system integrates with file coordination and other advisory locking mechanisms to coordinate with applications.

With these three mechanisms, we are able to sync files from the cloud and keep them in sync in case of remote changes. The last flow deals with syncing up local changes to the cloud. The system detects when local items have changed and calls a modifyItem method on your extension, passing in the exact set of fields that have changed. It aggregates low-level events into events that are meaningful for sync. For example, the kernel detects safe saves and remaps your item identifiers to the new file IDs transparently. The system will also zip package files for you if you request and present you with consistent package-level changes.

In response to the modifyItem call, you will update the state of the item server-side. If the contents of the file have changed, the system will hand you a clone of the changed file so you can upload a consistent version even in the event of further changes.

When done, you call a completion handler. The completion handler is used to update the version identifier of the item and confirms delivery of the change to your extension. The completion handler also takes the final state of the item as a parameter. Updating an item in the cloud may change its state, for example, if it conflicts with a remote change. Since you pass the final state back, the system is able to update the local state of the item to match the truth in the cloud.

There is a fifth flow: eviction. The system will evict local files automatically and without involving your file provider extension when there is an urgent need for disk space. That might happen, for example, when the user is recording a video or downloading a software update. The system will evict the minimum set of least recently used files necessary to free up the disk space required to write those new files. Let’s review the transitions. Eviction turns a local file into a dataless file, and download turns a dataless file into a local file. Files can start dataless if they are created remotely or not if they are created locally. But not all files can be evicted. The system will only evict a file that you report as uploaded so that it can be downloaded again. So there really are two sorts of local files: uploaded and non-uploaded files. After a local edit, the new version of the file needs to be uploaded, so we’re back in a non-evictable state.

In this presentation so far, we have seen how your file provider extension is invoked by the system to download files upon access and to upload files after local edits. While you’re not involved in disk-pressure triggered eviction, there are methods to trigger or prevent eviction from your extension.

This was a lot of theory. Let’s have a look at one of the flows in practice.

We’ve written an app that runs a small local file server and embeds a file provider extension that operates against that server. It’s called FruitBasket. I’ve already logged in to that server, so there’s an entry for the root folder here in the sidebar. I’ve also selected the root folder which caused the system to make dataless entries for the items in that folder. You can tell that the items are dataless from the cloud download icon next to the file name.

We’re going to use ‘cat’ on the command line to read a file. Since the file is dataless, this will cause a content fetch in our extension. I’ve already attached to the extension in Xcode and set a breakpoint to intercept this content fetch.

‘cat’ is running, and our breakpoint has hit. Since we are blocking the completion of the content fetch, the read in our Terminal window is also blocked. Note how in the Finder window, the cloud icon has been replaced by a progress indicator. The system has a consistent view of the download status. Of course, since we are actually blocked in the debugger rather than busy downloading, the progress isn’t updating.

Let’s continue. I’ve set a second breakpoint just before we call the completion handler.

At this time, our provider has downloaded the contents of the file to a local URL on disk. Once we call the completion handler, the system will swap out the contents of the user-visible file with what we’ve downloaded. Let’s unblock the system by continuing in Xcode. The status in Finder updates to show the file as being local, and the read that the cat process was blocked on succeeds.

I’ve still got the breakpoints set, but now that the file is local, I can run ‘cat’ again without hitting the breakpoint. Those reads are going against a regular local file and don’t involve our extension.

Of course, this is just a small part of the feature set of our sample file provider. We’ve covered the full feature set of the API, and we are publishing the source code as part of this session. Let’s talk about how you can approach implementing the flows that we’ve talked about. First of all, we’re going to want to tell the system that we’re ready to sync. This will make an entry show up in the sidebar in Finder. We call these entries domains, and they usually correspond to a login session on your cloud server. Each domain has a unique identifier, and to make it show up, you create a new instance and add it via the manager object. You can also remove a domain. You’d usually do this when the user logs out, but it’s also going to be useful during your initial development and testing. With the domain showing up in Finder, the system will request enumeration of items as soon as you navigate to the entry. So let’s implement that next. Our first step here is to implement an item class. Its instances represent the individual entries that we’re going to enumerate. Then we implement an enumerator that calls the system with our items when the system requests it. At this point, we can look at directories by navigating to our sidebar entry. Of course, all the files in those directories will be dataless. Let’s change that by implementing content fetch. The fetchContents method is called by the system when we open one of the dataless files. Our job is to download the file contents to a location on disk, then call the completion handler with that location’s URL. The system will use the contents to fill the dataless file and then clean them up for us. To allow our directory structure to stay in sync, we’ll implement another type of enumerator. This one syncs remote changes. The system calls the currentSyncAnchor method first to get a sync anchor. You return a data object that describes a change cursor for your database. Whenever you signal that something has changed, the system will ask for changes since the last anchor you provided. You can then return the changes and finally a new anchor. The last step is to allow sync up of changes. If the system detects changes to the local files, it will call one of three methods to create, modify, or delete an existing item. We’ll have a quick look at the create method. The system hands you the new item that it asks you to create. This is a system item, although it follows the same protocol as your own items. The system will also hand you a set of fields that are of interest on the item. For example, the item may or may not have extended attributes attached to it, and there are fields to describe that. If the content field is set, the system will pass you a file URL with the contents. Items that describe folders or symlinks will not have contents. Your job is to upload the new local item’s data to the server and then call the completion handler with the resulting remote item.

And that’s it. At this point, you have a functional file provider on macOS providing files on demand, propagating local changes to the cloud and remote changes to the Mac. There are a lot of additional optional APIs in the File Provider framework, which allow finer integration with the system. Let’s have a have a look.

Icon decorations can be used to visually decorate items in Finder. You can badge a file icon, emboss a folder, or indicate sharing status. You provide custom artwork for the decorations via a UTType declared in your app.

Contextual menu actions allow the user to execute custom actions on your files via the contextual menu. There are UI and non-UI variants. You can define which files these actions apply to with NSPredicates declared in your extension’s Info.plist.

Pre-flight alerts allow you to warn the user before they take an action which may have unintended consequences. The alert UI and the criteria to activate the alert are configured in the Info.plist as well.

So what are your next steps? Well, first of all, you can download the session’s sample code. It’s very comprehensive, and it’ll give you a lot of pointers. Add a target to your existing app for the new extension. There is an Xcode template that will help you get going. From there, all you have to do is implement the method stubs in the order we’ve discussed, and you’ll be up and running in no time. Thank you for watching this session. We look forward to seeing your file provider extensions on macOS.