Using Core Data With CloudKit

Good morning. My name's Nick Gillett. I'm an engineer here at Apple on the Core Data team, and it is my pleasure to welcome you to Using Core Data with CloudKit.

Today, we're here to talk about a belief that I have that all of my data should be available to me on whatever device I have, wherever I am in the world. And to make this a reality, we have to make it a lot easier for you to add this functionality to your applications.

I'm sure many of you came in today with an iPhone, and you may also have a Mac at home. You may even be toting around a MacBook or a MacBook Pro in your backpack while you're here at the conference. And the data that we create on all of these devices is naturally trapped. Right? There's no easy way for us to move it from one device to another without some type of user interaction. Now, to solve this, we typically want to turn to cloud storage. Because it offers us the promise of moving data that's on one device seamlessly and transparently to all of the other devices we own. And cloud storage has benefits even if we only happen to have a single device. Right? As our applications create data on this device, it can be backed up in the cloud and stored. So that when we get a new device, whether by choice or by accident -- ask me how I know -- on the way out of the store, that device can be restored to the one that we always knew and loved. And it may surprise you to learn that there are some existing technologies on our platforms that can help us with this problem. For example, Core Data provides a robust set of APIs for managing data locally in your applications and on disc. And the CloudKit framework provides access to one of the world's largest distributed databases. Both of these frameworks are available on all of our Apple platforms. And because of that, enable us to build a wide variety of applications.

In fact, these frameworks are actually quite similar. They even express themselves using a common set of patterns and paradigms. Modeling their APIs in terms of objects, models, and stores.

Now, in Core Data, we call these objects instances of NSManagedObject, and they're what gives our application access to values that we store on disc.

CloudKit similarly exposes a CKRecord, which is a key value like store for accessing data that you've stored in the cloud.

These objects are described by what we like to call a model. And in Core Data we call that an NSManagedObjectModel, which you can create in code or using the model editor in Xcode.

Quite similarly, CloudKit uses a Schema. And the CloudKit schema can be defined either by CloudKit dynamically as you use CKRecords in the development environment or using the CloudKit dashboard.

Finally, objects are persisted to use the Core Data vernacular, in what we like to call a store. In Core Data those are instances of NSPersistentStore. But in CloudKit, CKRecord's are stored in a CKRecordZone or in a CKDatabase.

And so, as you can see and indeed as many of you have pointed out to me over the years, it would be great if we could make it easier to combine these two conceptually similar frameworks. And so to show you just how much easier we've made that this year, I'd like to take you through what it's like to create a brand new application in Xcode. Here you can see, I have Xcode open. And I'm going to create a new iOS project as a Master Detail application. I like Master Detail applications because they give us a great UI to build on top of when exploring features of Core Data. So, I'll select it and click Next.

And then give my application a name. In this case, just WWDC Demo. And, because we're here to hear about Core Data, we'll check the Core Data checkbox. New in Xcode 11 is this checkbox called Use CloudKit. And this tells Xcode that we want to generate an application that's designed to use both Core Data and CloudKit. So, let's check that, and then we'll click Next and find our application somewhere to live on the file system.

After we tap Create, Xcode generates the application for us. And if you've ever built a CloudKit application before, you'll know that there are a couple additional things we need to add to this before it's ready to build and run. These come in the form of Capabilities, which we add using the Signing & Capabilities tab. We need to add two. The first is the iCloud capability. So, I'll add that by clicking this plus (+) next to Capability, type iCloud, and hit Enter.

When I do that, I can hit the CloudKit checkbox and you'll see that that adds push notifications for me as well. Xcode has also automatically created an iCloud container identifier for my application to use. Next, we want to add a background mode capability. And to do that, I'll hit the plus (+) again, and type Background, and hit Enter. And the reason we do this is to enable remote notifications, which allow our application to receive push notifications when it's not running.

So, let's run this application and see what Xcode has created for us.

Here you can see that we have a very simple application with two view controllers. A table view on the left and a detail view controller on the right.

And I can add some data to this application using the plus (+) button in the upper right-hand corner. By default, Xcode generates a very simple Core Data data model for us that's just a simple timestamp.

But, we're here to see what it's like to add sync functionality. And to do that, we need another device. So, let's run the application on our iPhone. And you can see that we have the master view controller and all of the data that we added on our iPad. Now let's add some data using our phone. And of course, watch that sync back to the iPad. Now, because I have a magic push notification in this controller, I can make this happen whenever I want.

But let's see something that's a little bit more realistic. I'm going to delete all the data from this application that was added on my iPhone from the iPad. These top four rows. And then, I'll do the same thing from the iPhone, deleting all the data that was added from the iPad.

Now, previously I used a magic push notification. But I'm not going to touch the controller anymore. I'll just let this video play so that you can see what it was like to actually watch these two devices converge when I filmed this.

Right? Now, as contrived as that might be, it's pretty awesome that in a few simple clicks we've built an application that syncs end to end using both Core Data and CloudKit. And I'd be doing you a disservice if somewhere in the application delegate I'd hidden 15,000 lines of code. So, let's see what that looks like.

Now, this is a pretty standard application delegate. In fact, if you've ever built a Core Data application before using Xcode, it'll look very familiar to you, including this area where we set up the Core Data stack. The only thing that's different about this application is some new API in Core Data this year called NSPersistentCloudKitContainer, which is designed to help you manage Core Data stores that are backed by a CloudKit database.

Now, if you've ever built a Core Data application before using Xcode you will have seen NSPersistentContainer here instead, which is NSPersistentCloudKitContainer's superclass. Because of that, you can add CloudKit functionality to your existing Core Data applications by changing as little as one line of code. What exactly is NSPersistentCloudKitContainer? Well, it's an encapsulation of a set of really common patterns that we saw everyone have to build when they wanted to implement end-to-end sync using CloudKit. And it's designed to save you thousands of lines of code.

It's also a foundation that we hope to be able to build on with you iteratively in the years to come. And of course, to do that -- get ready, here it comes -- we need your help.

We're going to need feedback about how NSPersistentCloudKitContainer works for you and what features it's missing, or whether or not its existing features are enough to meet the needs of your applications. So, let's look at a little bit -- a few of those features in detail.

NSPersistentCloudKitContainer provides your application with a local replica. A complete mirror if you will of the Core Data -- or the CloudKit database that backs it.

And, it also implements a robust scheduling and error recovery event loop so that your application doesn't have to worry about any operations. Finally, it handles the transformation between instances of NSManagedObject, that I mentioned earlier, and CKRecord.

Now, a local replica is important for a number of reasons.

But it means that as your application works with objects, it will be writing them to a local store file managed by Core Data. And, it will be reading them from that file as well. And this is because of the difference between the performance profile of the local database versus cloud storage. Right? When we think in terms of latency of accessing a local file, we can read from a file on disc in milliseconds at the worst case. Whereas, over the network, it might take us seconds or minutes to fetch the data that our application needs. Likewise, the local store file can provide your application with much higher bandwidth, measured even on our iPhone devices in gigabytes per second, whereas the cloud is sometimes limited to megabytes or kilobytes per second of available bandwidth.

Now, this local replica necessarily adds a significant amount of complexity, and that's why NSPersistentCloudKitContainer implements a robust scheduling and error recovery event loop for you. So that as your application writes data to the local store, NSPersistentCloudKitContainer automatically moves those objects up to the cloud.

And, when something changes in CloudKit, NSPersistentCloudKitContainer will schedule work on the system to bring those objects down and import them into your local database, making them available to your application.

Of course, during this process, your objects are necessarily transformed from instances of NSManagedObject into instances of CKRecord by NSPersistentCloudKitContainer.

And likewise, when something changes in the cloud, those CKRecord's will be made available to you as instances of NSManagedObject in your local store file.

And that's what NSPersistentCloudKitContainer brings to your application. It's a complete local replica of everything in the private database in CloudKit. You should know, though, that we do manage a specific custom zone for Core Data syncing. We implement automatic scheduling so that you don't have to worry about optimizing any operations or scheduling them in the system. And, I think more importantly, you don't have to worry about implementing any error recovery logic in your application.

Finally, we implement automatic serialization from NSManagedObject to CKRecord. And we use your NSManagedObject model to figure out how to do this.

But it'd be obtuse of me to stand up here and tell you that once you've adopted NSPersistentCloudKitContainer, you're done, and you have a fully functioning application. So, I'd like to spend the rest of the talk thinking about what it means for you to build on top of NSPersistentCloudKitContainer. And I think that starts with building great applications that are powered by Core Data.

Later, we'll also look at how you can extend the foundation that we've built into NSPersistentCloudKitContainer to customize it to your needs.

Now, for me, building great applications with Core Data starts by absorbing a lot of knowledge. And, to that end, we've written a ton of documentation this year about how NSPersistentCloudKitContainer works and how you can integrate it into your applications.

There are some features of Core Data that I feel go really well with NSPersistentCloudKitContainer. Things like the FetchResultsController, which helps you build scalable user interfaces backed by a significant amount of data.

And query generations which help you stabilize those user interfaces against changes that might be happening in the background.

Like say from NSPersistentCloudKitContainer.

And finally, there's history tracking, which we introduced a few years ago to help you understand what's changed in the database. And with NSPersistentCloudKitContainer, you can use it to decide whether or not any of those background updates are relevant to what your user's currently doing.

We'll be covering these and a whole lot more in our session on Thursday at 3 pm. And, to go along with that, we're also introducing a new sample application this year that's designed to give you something to hold in your hands, to feel how NSPersistentCloudKitContainer works along with all these other features of Core Data. It's designed to manage a set of posts. And posts are a theme that we've been building on over the last few years in Core Data.

Right? They're a great object for helping us understand how different pieces of your object graph will be affected by CloudKit. Here you can see that our data model is pretty simple. Right? We have a title and some content. And then, we have a set of tags that we can associate to each post. The application will even let you see what it's like to manage photos with CloudKit. Allowing you to attach files from the device's Photo Library to a post. Now, let's talk about what it's like to build on top of NSPersistentCloudKitContainer. And, as you might expect, this is a fairly dense section of the talk. But you should know that our documentation covers in more detail a lot of things that we're going to talk about today. So, don't worry if some of this goes right out the window. There are a number of ways that we see clients internally extend NSPersistentCloudKitContainer, beginning with working with multiple stores.

And, we also see that clients like to customize the schema that they've been using with CloudKit, and indeed, the one that we use with NSPersistentCloudKitContainer. This of course, as you may know, is because CloudKit is available on a number of different platforms, not only at Apple but also using Web Services or JavaScript. And so, you should be able to work with NSPersistentCloudKitContainer's objects even if you're not doing so on one of our platforms.

Finally, we'll take a look at Data Modeling for collaboration.

Now, there are a number of great reasons why we might want to use multiple stores in our application. Especially when working with network backed stores.

Multiple stores can help us segregate our data between distinct use cases in our application. And, they can help us provide different types of constraints.

Right? So, if we wanted to have one store file that manages a very specific set of validation constraints, maybe to validate user input, we can use a separate store for that.

Multiple stores are also a great way to throttle or coalesce data that's written very frequently on device.

And this can happen when you're reading something from the device that's generated maybe by the device itself or by an algorithm that you have which creates data. If this algorithm generates data at a very high rate, it can be very expensive to constantly sync all of that data up to CloudKit. And so, we see clients inject another store file into the mix and use that to coalesce this data until its ready to be analyzed and then uploaded to CloudKit later.

To show you how this works and how Core Data can make this easy for you, we're going to leverage a feature of NSManagedObjectModel called Configurations. And here you can see we have our sample apps, NSManagedObjectModel in the model editor in Xcode. Now, let's say that I want to add locations to the post. Right? I want to record the location where a post was created whenever that happens. But locations can be generated at a high rate by the system and I don't need them unless a post is actually being created at that location. So, let's separate them from the rest of the data model.

I'll start off by adding a new entity by clicking this plus (+) in the lower left-hand corner to store my location information. Now, my location's going to be pretty simple. It's just going to include a latitude and longitude that are both doubles that are exposed to me by the Core Location framework. Of course, yours may include other things like altitude or accuracy.

Now we want to segregate these locations from the rest of our data. And to do that I'll create a new configuration, again, by hitting this plus (+) button, but this time holding down my click to expose a menu that allows me to add a configuration.

I'll call this configuration Cloud and add all four of the entities that I actually want to sync to it.

Here you can see that now the Cloud configuration has just the entities that I want to be synced with CloudKit. Let's create a new one called Local that will store our location objects.

And, in just a few lines of code, we can put this to work for us, empowering Core Data to tell our stores automatically what types of objects they stored. At the top you can see we create an instance of NSPersistentCloudKitContainer, and then, we leverage something called NSPersistentStoreDescription, which tells NSPersistentCloudKitContainer about the types of stores that it's managing.

We create an instance of NSPersistentStoreDescription that points at the file called local.sqlite to hold our location information. And, we assign it the configuration of local that we just created.

Then we set up our cloud store. Similarly, we create an instance of NSPersistentStoreDescription and we point it to a different file; cloud.sqlite. Then we give it the configuration Cloud, which tells NSPersistentCloudKitContainer that only the entities like post, and tags, and attachments, and image data should be stored in the store.

Finally, we assign it an instance of NSPersistent CloudKitContainerOptions, which tells NSPersistentCloudKitContainer what iCloud container identifier this store should be synced with. Last but not least, we assign these two store descriptions to the PersistentStoreDescription's property of NSPersistentCloudKitContainer.

And with NSPersistentCloudKitContainer we can take this even further.

We already have local store and a cloud store. But what if we want to share some data in CloudKit across multiple applications we happen to work on? Well, NSPersistentCloudKitContainer supports that as well. In fact, because you're using Core Data, your application will easily be able to handle all the data from these stores at the same time. And, Core Data will automatically help you write data that you insert into the correct store file automatically.

We can do this by just adding three lines of code. Right? We create a new StoreDescription that points to our shared store file and give it a new configuration that we might create called shared.

Finally, we assign it a new instance of NSPersistent CloudKitContainer Options that identifies the container we want our shared data to be stored in; in this case, iCloud.com.wwdc.shared. And of course, last but not least, we assign it to PersistentStoreDescriptions. Now let's talk about the schema.

And, I want to cover a few points about the schema that I think are sort of the highlights. The most important things that you need to know when you're looking at the records that we create in CloudKit.

I'll start off by talking about how we manage record types and how we manage entities, which are what you create in your NSManagedObjectModel.

Then, we'll look at how we implement a feature called Asset Externalization, which allows you to seamlessly store arbitrarily large values in CloudKit using NSPersistentCloudKitContainer. Finally, we'll talk about how we manage relationships and how that might be different from the CloudKit experience you're used to.

To do that, we're going to use the ManageObjectModel for our sample application. And I'm going to start off by focusing on the post entity. You can see that it has two attributes; a title string and a content string. And it also has two relationships to the attachment and tag entities.

Core Data generates an actual class for you to use in code as a subclass of NSManagedObject that looks like this. And you can see that all of the attributes and relationships are represented on that class. Now, this is the record that we will generate in CloudKit that goes along with this post.

And these are some example values that I've used to populate this record and make it what we call fully materialized.

Now, there are some things that I want to highlight to you first. The record ID. Core Data owns the record ID for all of the objects that it creates in CloudKit. And, for each one, we will generate a simple UUID to use as its record name. When the Record Name is combined with a zone identifier you get a CKRecord ID. At the bottom, you'll see how Core Data manages type information. And, there are two interesting things here. The first is, what the heck are all these CD under bars? This is Core Data's way of segregating the things that it manages to be separate from ones that either CloudKit implements for you -- you wouldn't believe how many people add modify date to their CKRecord.

Or, ones that you might add yourself. And so, we prefix everything; the record type and all of our field names with CD under bar.

But, in the CD entityName field, we keep the real entity name of the object that this record goes with.

And we do this so that we can implement a feature called entityInheritance, wherein you could have subclasses of a post, let's say. Perhaps an image post, or a video post. The actual entities will always be identified by the CD under bar entityName field.

And we do this so that you can implement CK queries that get at all of the entity hierarchies you're interested in by querying a single record type.

Now let's look at how we implement these two strings. Because these are variable length fields and our translation of them to CloudKit has some interesting quirks.

You'll see that we've inflated them to four total fields. And the reason for this, is that this is how we implement asset externalization.

Here you can see that we have both a CD under bar content field and a CD under bar content under bar CKAsset. This allows us to store strings that are arbitrarily large. Anywhere from simple kilobytes all the way up to hundreds of megabytes or even gigabytes in length.

And when I said that this record is what we call fully materialized, what I mean is that you won't ever actually see all four of these fields at the same time. If the strings are all very short, then you'll just see CD under bar content and CD under bar title in the record. But, if one of them grows to be very large, approximately larger than 750 kilobytes, or if the total size of the record exceeds CloudKit's maximum 1 megabyte limit, you'll begin to see asset fields, or in this case, CD under bar content under bar CKAsset in the record instead.

If you're consuming our records on your own, then you'll need to check both places to see whether or not a value has been set for a specific attribute.

Now let's take a look at the relationships on a post. You can see here that they're both instances of NSSet in the object that Core Data generates for you to use in your code. And this is because a post has to-- what we call too many relationships. That means that a post can be assigned too many attachments or it can have many tags associated with it. The attachments relationship is what we like to call a Many-To-One, because an attachment can only be assigned to one post.

When we generate code for this, you'll see that there's an NSSet on the post, but there's only a single post object declared on the attachment ManageObject.

And this is the record they will generate for an attachment. You can see that it has a UUID, an entity name, and a record type just like a post does, but you'll also see an additional field called CD under bar post. This is how we store the relationship for a Two-One relationship.

The UUID of the related record in CloudKit will always be stored on the object it's linked to. And, you might be wondering why we don't do this with CKReference instead. And that's because CKReference has some limitations that we don't think work well for Core Data clients. Namely that it's limited to 750 total objects. But by storing a relationship this way, you can hold as many will fit in your CloudKit container. Now let's look at a Many-To-Many relationship. In this case, the one between a post and its tags.

You can see that there are two NSSet's on this object and this is because both objects can be related to many of the other type. But when we generate the records for these objects, there aren't any fields that we materialized to hold this relationship.

Instead, Core Data materializes a custom join record. Now, if you're familiar with relational databases, you'll recognize this concept. It's basically an extrapolation of a row in a join table. And it's called a CDMR or Core Data Mirrored Relationship.

A CDMR contains a set of twoples that are designed to describe the two objects that are linked, beginning with the entity names of the two objects that are linked, and their record names. Now, as I said before, this is not the record ID. This is the record name that you have to combine with a zone identifier to get the identity of the records that have been linked by the CDMR. Finally, we also encapsulate the exact relationships that were used to make this link.

So why am I spending so much time talking about relationships? Well, they have a huge impact on how we model our data for collaboration. And I need to be very clear here, collaboration is not conflict resolution. Conflict resolution is implemented automatically by NSPersistentCloudKitContainer using a last writer wins merge policy.

And the reason we do this, is that the job of conflict resolution is to keep the object graph and the data in CloudKit consistent with how you've modeled your data.

But, over the years we've heard that this can be a little frustrating. So, let's take a look at how you can implement better merge behavior and align that merge behavior with your specific customer use cases using relationships.

To do that, we're going to use our post application again. And I'm going to create a post, but I'm not going to assign it any content.

I'll let it sync over to another device let's say, and then I'll make some edits on each device at the same time. Now, this is what we would traditionally term to be a conflict. But, actually, this is a great example of two devices attempting to collaborate on a value.

Now, NSPersistentCloudKitContainer seeing that something has changed in CloudKit while it was away, will resolve this to preserve one of the two values. Right? Using a last writer wins merge policy. Which means that we'll either get collaboration is great or everyone should do it. And it's true.

Of course, you might be thinking that -- well, this is kind of dumb. Why don't you just concatenate the two strings together, Core Data? And we could. Except that if you took our advice over the years and implemented incremental saving you might end up with something like this, which is not what anyone expects. And it's not even English.

So, how can we do better? Right? Clearly this is a problem that's very important to us and it's very dear to our customers.

Well, let's look at our post entity and see if there's some changes we can make that would make this better. The first thing to do is to stop creating collisions on flat values. Content is simply a flat string and there's nothing that we can infer about the merge behavior that you want for that string, simply from looking at it.

But, if we break it up as a relationship, multiple devices can contribute contributions to the content field all by themselves. And because of the way we store Two-One relationships, many devices can contribute these objects at the same time without creating a collision on the post object itself.

So, here I've broken it up as a very simple post content entity with a simple string.

Now, because of how we store Two-One relationships, these will be combined by devices on their own. Right? We need to traverse them in order to assemble the final value. And we call this eventual consistency. As both devices contribute post content objects, they'll download the post content objects from the other device and concatenate them or merge them, or whatever topology you want to apply to this problem to assemble the final value.

But we want that to be the same across all devices and a difference in download order could create a different final value. So, we can order them say with something as trivial as a date. In this way, the devices can order the post content objects before they do a concatenation using a simple sort descriptor in Core Data, giving them a consistent ordering and merge behavior across all devices.

But, I'm a bit of a distributed systems nerd and time is evil.

In fact, a devices notion of time might not even line up if they're in your house together.

So, we can take this a step further and implement a full causal tree by leveraging a relationship to a parent contribution and giving the entity some information about the device that's making the actual contribution.

In this way, we've implemented a rough sketch of something called the conflict-free replicated data type. And this is an awesome emerging field of computer science that allows us to deploy algorithms to create consistent merge behavior across different user-user scenarios. And that has been Using Core Data with CloudKit. It's been my pleasure to show you NSPersistentCloudKitContainer and how you can easily implement sync functionality in your applications with it.

We've also got some great new sample code and documentation for you this year that I hope gives you a great experience with all of these new APIs. And finally, I can't wait to see what you build with NSPersistentCloudKitContainer and how you extend it. We'll be in a ton of labs this year.

Every day this week. And as you know, we have a talk on 3 o'clock that covers a lot more features in Core Data in Thursday.

Thanks a lot, and I hope you have an awesome WWDC. [ Applause ]