What's New in Apple File Systems

Good afternoon, ladies and gentlemen. Welcome to the Filesystem session. In this session, we're going to cover several topics related to the filesystems. First, we're going to have a look at the role a filesystem plays in protecting system software on macOS.

We're going to describe how volume replication works in APFS. And finally, we are going to talk about the new and very exciting feature coming to the devices running iOS and iPadOS, the access to the external files on the external media. But before we start, let's rehab what happened to APFS recently. APFS has been the default filesystem it -- on iOS and tvOS since 10.3 and on macOS since High Sierra.

One of the features which APFS introduced is a built-in volume manager.

The concept of a volume is not new to APFS. They existed in HFS Plus.

The HFS Plus volume is a one-to-one map to -- on this partition. And it occupies contiguous range of blocks on disk.

Because of this one-to-one mapping to partition and HFS Plus volumes cannot be added easily. In order to add a new volume to partitioned disk, you need first to shrink the existing volume. And then you can add a new volume. APFS volumes are a lot more flexible because they share their own disk partition space with their siblings.

And that flexibility allows higher levels of a system to implement features which would not otherwise be possible to implement with the old-style volumes.

One of us -- one of the features which we'll implement is protecting system software from malicious or accidental updates.

You may remember back in the macOS El Capitan days, we introduced the concept of system integrity protection.

It was used to control access to a part of the directory hierarchy.

Some directories would be write protected, while changes would be allowed in the other parts of the filesystems. This year, we're taking this one step further. We're making entire root filesystem read-only.

Obviously, a system where you cannot install new software or where user cannot save the data is not particularly useful.

In order to understand how we can reconcile those two conflicting goals, read-only and writeable at the same time, let's look at what's happening when we upgrade from macOS Mojave to macOS Catalina. A typical container on macOS Mojave will have one main volume, and few service volumes. For example, VM. The main volume is used to store both user's data and system software.

When we start upgrade-- When we start the upgrade to macOS Catalina, we first change the role of the main volume and mark it as data volume.

We then can prune some parts of the directive hierarchy which contains only system software.

Once this is done, we can move to the next stage where we create a new empty volume, which will be used only to store system software. We'll populate that volume with the system content. And once this process is done, we can declare victory on the protection front. We are read-only. We're protected. It's good, but it's not enough.

We still need to somehow tie the new system content with the user content. And for that, we introduce the concept of a volume group. A volume group is one data volume and one system volume. And they treat it as it as a single entity.

The UI present it as a single disk. They share encryption state. If your volumes are encrypted, then the same password can be used to unlock both volumes. Almost everything looks as a single unified entity.

There is one thing which is missing. We still need to somehow provide an illusion of a single directory hierarchy. Traditionally, it was done by mounting filesystems on top of directories in the root filesystem.

With a number of crossing points, which we need to introduce, and the number of volumes which are required in the filesystem, that approach becomes rather expensive. So to deal with this, we introduced a new concept called Firmlink. The Firmlink is a new filesystem object.

It's similar to Symlink. But unlike Symlink, Firmlink support backwards and forwards path reversals, which is consistent in its representation. And that consistency is rather important. If you ever had to deal with an application, which absolutely insists it has to live in a particular directory, for example, such as applications. You know that you have to be able to walk both from the top of a filesystem from the road down to the leaves and backwards and get the same path. We can do that with Firmlinks.

The Firmlinks are a traversal point from a directory on the system volume to a directory on the data volume.

They only have one-to-one mapping. One source. One target.

You cannot use a Firmlink to cross a boundary of a volume group.

The Firmlinks are rather transparent to the user and the application. They are created by the install at the installation time. And they are not supposed to be noticed.

Once we get this new tool, we can use it to stitch up volumes together. The installer will create entry on the system volume.

And point them to the corresponding volumes on the data volume.

And once it's done, we have a unified directory hierarchy. We can reboot, mount root as read-only, and enjoy the protection which it gives us.

Life is good. Everything's protected. Everything's running. But you still have to remember that the volumes are split during the install process. There is no way to avoid it. It will happen. In the developer's preview, we elected to leave the root filesystem writeable to make it easier for you to test your application. If you want to mimic your behavior, which will be implemented in the future, you can create a special file in the root directory and on reboot, your volumes will be mounted read-only.

That will change in the next build over next seed.

In the release build, you would still be able to mount root filesystem as read-write if you disable system integrity protection.

But this is not a persistent change. On the reboot, it will revert back to read-only state. You can remount it read-only, read-write again.

And again, it will revert to the same state on reboot.

As you can imagine, this is rather big change in the way macOS shapes and installs. And it could catch some applications. If, for example, your application uses a complex layout on the filesystem or comes with the installer package, make sure it works on the new read-only root partition.

If you're writing a backup utility, which cares about inode numbers, filesystem IDs, and the like, make sure that you test it, because the assumptions you may had previously may not be true. So the bottom line is test, test, test. And next, I will hand over to Jon Becker to talk about volume replication.

Thank you, Max.

Good afternoon. My name is Jon, and I'm going to be talking about volume replication with APFS. So let's dive right in. What is volume replication? Well, the basic idea is pretty simple. We'd want to copy one volume to another volume.

Sounds simple. But the important aspect of this is that we want the fidelity of this copy to be as high as possible. In general, it's not sufficient to just be doing file-by-file copies. We want to be copying all volume content. All data, all metadata, volume attributes. If the source volume contains a bootable OS, we want to be copying the metadata that makes that volume bootable so that the target of our copy will also be bootable. Now I'm going to be talking about replication in the context of the Apple Software Restore command line utility, or ASR. ASR has been around for a very long time. Many of you may be familiar with it. And its primary function is to do volume replication.

Now, with ASR, it's very common for the source of our replication to be a disk image. And in that context, we often refer to the procedure of doing this replication to occur -- to a target volume as restoring, hence the name Apple Software Restore. But you'll hear me use the terms restoring and replicating pretty much interchangeably.

So who wants this? Who needs this functionality? Well, if you are in education or Enterprise IT, if you do things like set up large labs with lots of machines, or if you write backup utilities, then replication is a function that you probably need or want to use with some regularity. As we'll see, some of the new features of APFS present a challenge for how we can do replication.

On the other hand, as well also see, these new features also present an opportunity to make replication a more powerful and flexible operation. So I want to back up for just a second and talk about how replication has looked in the past prior to APFS. Now as Max demonstrated before, with previous filesystems, like HFS Plus, the filesystem, or I should say the volume, is in a one-to-one relationship with the partition that contains it. And what that means is that the volume is backed by a contiguous block device. So replicating can really involve just doing a block copy of the entire partition. Of course, if we're copying the entire block device, we are copying all of the filesystem information in that volume. Now there may be some complications like as in this diagram. We have the source and target partition not being the same size. But there are ways that we can fix that up. And so by and large, block copying is a very effective and relatively simple way of doing replication. But of course, with APFS, there are some new features that complicate this picture.

So APFS, as Max told you, has some features, like volume management and space sharing. And as we can see, if we take a look at Volume 1 in the diagram here, it's spread out around its container partition. And so it is not a contiguous block device.

And it may be -- its data may be interspersed with the data for another volume in the same container, like Volume 2 in this example.

Furthermore, we of course care about security and privacy. And so we have to think about encryption. Now with APFS, encryption is done at the filesystem level. And what's more, on Macs that have the T2 security chip for internal storage devices, that encryption is on all the time. And it is tied directly to the hardware, meaning that it is specific to that storage device in that Mac. And so if we would try to copy the blocks for that volume and take them somewhere else, they won't be decryptable. So the upshot here is block copies are really not a possible way to do replication with APFS volumes. So how do we reconcile this with the needs that we have? Well, new with macOS Catalina we're introducing APFS volume replication with ASR. ASR and APFS. Yeah, thank you. ASR and APFS are tightly integrated. And ASR can have APFS generate a stream from the source volume. And that stream then gets written to the target volume. Now because APFS is generating this stream, it knows where it needs to read the parts of the source volume.

And so it's not a problem that that source volume is not a contiguous block device.

As far as encryption goes, if the source of volume is encrypted, then it will be -- the data from it will be decrypted on the fly as the stream is generated. And of course, if the source volume is protected by FileVault, then it does need to be unlocked by user action prior to this replication taking place. If the target volume is itself encrypted then the data is in -- or the data is encrypted as it is written from the stream to the target volume. And so in this case, that data is encrypted from the get go by the time it hits the target volume. Okay? One other nice feature of this replication is that as the stream is generated, the volume data is defragmented on the fly. The metadata is compacted, and so the resulting stream, and therefore, the resulting target volume, are very nicely optimized. This can be a great way to do -- to master images, for example. If you're mastering an image, and your final step is to do a replication operation, then your image volume will be very nicely optimized.

So when we do restores, replication, however you want to say it, there are a number of options that we can use. I just want to call out a couple of them. The first one is really very much like restores as we're used to. We can specify a source volume. You can specify a target volume. And the idea is that the target volume will be completely erased and replaced. Or its contents replaced by the contents of the source volume. Now in this case, we will have Volume 2 be our target volume. And so our restore would look something like this. You would see a sample command line right here.

And the result is Volume 2's contents are erased, replaced by the contents of the source volume. So the restored volume and source volume are the same. Notice, by the way, that in this example, the target container also has another volume in it. And that volume is left alone. Its contents are preserved. It does not form in any way a part of the replication operation. But there's another option that we have, which is instead of having to specify a target volume and erase that, we can instead generate a brand-new volume to be the target on the fly. And we do this by specifying the entire container as the target. And this tells ASR that what we really want is to generate a brand-new volume and restore to that. You can see another sample command line. And the result is a new volume is created and restored to. So in this case, Volume 1 and Volume 2 are both left alone.

Now I want to step away for just a second from replication. And I want to talk about snapshots in APFS. So a snapshot is just a point-in-time capture of all volume state. So for example, we may have a volume with a number of files on it. We can take a snapshot of that volume. And the result is a capture, a freeze frame, of what that volume looks like at the time that the snapshot is taken. If we make subsequent changes to the volume, like for example, deleting a file or adding some new files, the snapshot is still encompassing all of the state that existed at the time it was created.

So in this case, if we look at the live volume, it appears that that file that was deleted is not there. But in some sense, it is still there because it's part of that snapshot.

So you might wonder, "Well, what does this have to do with replication?" Well, once again, new with macOS Catalina, we can now do restores, replication of snapshots.

So to understand what that means -- Thank you.

To understand what that means, consider this volume here on the left. It has two snapshots in it, Snap 1 and Snap 2. They contain some files in common, the yellow ones, some files that are in one snapshot and not the other, some files that are in the other and not the one, and a file that's not an either snapshot. I can restore this volume to a target volume over here on the right. It's currently empty. And the idea there, of course, is that at the end of that restore, the target volume looks like the source volume. But instead of restoring the sort -- the live version of the source volume the way it currently looks, I can instead restore a snapshot. So if, for example, I want to restore Snap 1, I can specify that's the snapshot I want to restore. And the result is that my target now looks like Snap 1. And notice in particular that those two files, which were deleted from the source volume, have come back to life in the target volume.

Having done that, maybe I want to add some new files to my target volume. But then, maybe now I want to restore Snap 2 to that target volume. And of course, again, the idea is that at the end of that operation, the live target volume should look like Snap 2 from the source volume. But notice that both the source volume and the target volume already have Snap 1 on them. Wouldn't it be great, if instead of having to copy all of Snap 2, I could just copy the things that are different between Snap 1 and Snap 2? Well, indeed I can. We call that difference between two snapshots a snapshot delta. And the idea is when I restore a snapshot delta, I'm specifying the two snapshots whose difference I want to take. I perform the restore. And of course at the end, the live target volume looks like Snap 2 from the source. But there's three things that I want you to notice about this target volume. Number one, all of those files which were not part of the snapshot on the target have been discarded.

Number two, the files which were in Snap 1 but not in Snap 2 have been discarded from the live target volume. They, of course, still exist in Snap 1 on the target volume. And number three, the only things that we needed to copy were those files that were part of Snap 2 and not part of Snap 1.

Okay? Now this is a really powerful feature. It's a great way to do incremental releases. Imagine that you are updating a lab filled with 100 machines. You can save a lot of time, a lot of network bandwidth, if you're only copying the difference between a couple of snapshots on your source image.

So that's what I had to say about replication. Take home points here. A more sophisticated filesystem requires more sophisticated mechanisms for doing replication.

APFS volume replication is really best done using ASR because it provides the highest fidelity of copies. And it handles the encryption as necessary. And finally, we can now restore snapshots in snapshot deltas using ASR. And with that, I'm going to turn it over to Bill, who will talk about external file access for iOS. Thank you. Thank you, John. Good afternoon. You may recall two years ago, we introduced Files app and a new file provider, API.

Together, they offer an excellent experience for Cloud-based documents.

This year, we thought we could do more.

So this year, we're happy to announce support on iOS for accessing files on network shares and on USB storage.

For USB storage, we support everything from compact flash and CF and cards and sticks, up through USB raid boxes.

We support a number of filesystems. We support unencrypted APFS, unencrypted HFS Plus, and we support FAT and ExFAT. This feature is available on all iOS and iPadOS devices. It's available on iPad Pro with USB-C. And for lightning devices, it's available with the appropriate adapters.

Moving on for network shares, we support connecting to SMB 3.0 servers.

We support connecting over Wi-Fi, over cellular, and over Ethernet.

There are a lot of exciting features here. But one that we wanted to call out is for our iOS devices, iPadOS devices. We're supporting search using the Windows Search Protocol.

So all these devices can search your SMB server if it supports the WSP protocol.

We're also really excited to announce that that that includes the SMB server and macOS Catalina.

Before going on, I wanted to talk a moment about security. Security is a feature users have come to expect from iOS. Two of the tools we have used to help deliver this security are process separation and privilege separation. In developing this feature, we've kept this in mind and incorporated them from the ground up.

So for all of our volumes and shares, the filesystem manipulations happen not on the kernel but in a dedicated process space. This separation helps us deliver the security iOS users have come to expect.

Now, let's see it in an action.

All right, I have an iPad.

And I don't know if -- you probably can't see it. But I have a USB stick connected.

And I have mail. And in files, on the left, you can see the locations. iCloud Drive. If we had a third-party Cloud provider, they would be listed there as well. And we see a destination for our USB device. When we select it, we see our photos, our documents, all the files and directories on there. And we can manipulate them just like any other file in Files app. So to copy one, you just select it, get it ready for drag and drop, pick your destination, drag it, and drop it. And you can see it's already in the folder now.

Another thing we like to do with devices is let's copy photos onto them. Photos, I have a picture a friend took in India of tomatoes.

Let's save it to the USB. To do that, we just select the document, go to Share, go down the list to save to Files. You can see, as a destination, we have the USB stick listed. We just select it. It already is. We hit save. And it's copied. I expect many of you are application developers. And you're wondering how your application can take advantage of this. This feature is available to any and all applications linked on or after iOS 13.

So rebuild your application. And you can take advantage.

To see that in action, let's look at Numbers. When I open up Numbers, it's start -- it's starting with my iCloud Drive.

The USB is listed as a destination. We select it.

And there, we can see all of our documents. We can see our photos dimmed because they're not Numbers documents. And then we see that we've had two Numbers documents on this drive.

Let's open one of them. It's a loan comparison comparing two different loan amounts and two different interest rates.

Just for comparison, let's see what happens if we raise the interest rate.

No. Oh. Set it to 20, not 200.

Oh, no. Well, that's crazy. And you can see on the -- see live the interest rate amounts changing.

So what does this mean for you as developers? As I said, it's available if you're linked on or after iOS 13. So rebuild your application and test.

We're adding five different filesystem types to iOS. These filesystems act slightly differently than APFS on the internal flash storage.

One difference is iOS has always had case-sensitive filesystems. FAT and ExFAT are case-insensitive.

And HFS and APFS can be configured either way.

The Clone System call may not always be available.

So as these differences are -- if these are differences are important to you or as they are, please pay attention to volume capabilities. There are two different APIs or a couple of APIs that can get them for you. One I wanted to call out are the resourceValues in NSURL.

These can give you parameters for the filesystem you're working with.

Another important point is file movement may take time. So please put your temporary files near your working files. If you don't, right, this is helpful. Because Save-Save uses a rename at the very end so that a user always sees a good file. They either see the document that they started with, or they see the new save.

For this to work, we need that rename. And that only works if they're both on the same filesystem.

Also, if you're not careful about your temporary files, they may end up in your container on the internal storage. And that's going to lead to a lot of unnecessary IO.

File Manager can help you with this. If you ask for a URL for the itemReplacementDirectory appropriate for your documents, it will give you a temporary directory on the same filesystem.

Another thing is that external devices can go away.

A network can go out of range. A file server can go down. A CAT can disconnect a cable.

These things can happen. And your application needs to be robust in face of it. One thing I especially wanted to point out is mmap can be dangerous. It can be really powerful. But if the file goes away, the only way the kernel can tell you that is with a BUS error.

So one thing, if you're using NSData, there is a hint you can give NSData and say mmap this data from a file if it's safe.

The last point is that external devices, all of them, have higher latencies than APFS on the internal flash storage.

So if you're doing sizable IO, please keep multiple operations in-flight at once.

So to summarize today's talk, Max talked to us about how we're making the root volume read-only for enhanced security. John talked to us about ASR and how you can use ASR to replicate volumes, including snapshot deltas. And I talked to you about how we're adding support for external files to iOS and iPadOS and how that will let you access files on USB storage and network shares.

For more information, we have a lab after this session. And tomorrow, there's a talk, Combine and Advances in Foundation, where they're going to talk more about what you can do with Foundation on external media. And there's also a video session, What's New in File Management and Quick Look, that will talk more about the UI document aspects of this talk.

Thank you.

[ Applause ]