What's New in Core Data

Good afternoon and welcome to Session 220. My name is Rishi Verma and I'll be joined by Scott Perry later and we'll be presenting What's New in Core Data this year.

Before we get into what's new though, I'd like to tell you a little bit about what is Core Data.

Now a lot of you have made these amazing apps with beautiful UI and you've tied it all with data that you're either getting from an external data source or provided with your resource bundle. Now as you process those objects you're building up a complex graph and shuttling all those changes to your UI. And then as your user makes changes on the UI you're pushing all those changes back through your object graph and back to your data source. Well, Core Data makes this easy.

Core Data will manage your object graph for you.

Simply tell us a bit about your Cocoa model there and the object model editor, tell us a little about your objects, their attributes, how they're all related to each other and we'll take care of the rest and we'll also persist it in the back end of your choice, be it sequel light or your own custom store.

Now, as you ingest objects, your relationships will be changing and Core Data will maintain these for you, so if you set up any delete propagation rules in your object model we'll delete and object and propagate those deletes as you define.

Finding objects in your object graph is also particularly easy.

Simply use an NSFetchRequest and give it a predicate to find the objects you're looking for. And we'll go and find them for you.

Also convenient is batching.

This allows you to only pull up a smaller portion of the objects in your data set that may result from your fetch request, allowing you to have smaller chunks of data as you go through your data set and also another candy feature of NSFetchRequest is relationship prefetching. Tell us an object to fetch and we'll prefetch all of its related objects so when you traverse that relationship you're still doing so in memory.

Then you simply just tie this all to UI.

You take an NSFetch result controller and tie it to a Table View like we've done here. And as I delete the Apple butter, my UI will update accordingly.

And then as I ingest more objects and I add banana bread, my UI will update accordingly as well. This is all handled for you, you get all the key view behaviors and change notifications handled for you by Core Data. Excuse me.

Now there may be a scenario where your user is manipulating the object on the main context and on a background context you're ingesting the same new object and possibly updating the object the user currently is also manipulating.

This is introducing the multiwriter conflict.

In Core Data it has you taken care of here as well. We version all the objects and allow you to set a merge policy. If you do not set a merge policy we'll default to error when you save in your context and give you a conflict error allowing you to address the conflict as you see fit. Or you can choose from several merge policies that we have already provided, be it the in store memory persistent store trumps what's in memory or what is in memory trumps the persistent store. Choose what is best for you and your particular scenario.

Once you've adopted Core Data you get several benefits. I would like to give you two in particular that really are the best ones. An excellent memory scalability and aggressive lazy loading. What that means, it is we'll only load the objects you need when you need them into memory.

Adopting Core Data leads to a much smaller footprint, over 50 to 70% less code for you to maintain allowing you more time to go and work on new features for your app. Then you can join the over 400,000 apps already using Core Data in the App Store.

That's a brief overview of Core Data.

Now let's jump into the new stuff. All the new APIs we have for you.

First, let's start off with MS manage object and a new property called hash persistent changed values.

Previously you may have used hash changes, this was a rather basic dirty flag, if you touch the object, we would mark it dirty. But with hash persistent changed values we'll ensure that the properties on the object are different than what's in the persistent store ensuring you don't have any false positives.

Also new on NSManaged object is object ID's for relationship named.

This is ideal for working with large relationships mainly because we won't materialize the entire relationship in memory rather we'll return to typed array of object IDs to you. This allows you to go through these object IDs and work with your objects in smaller sizes. Let me show you a quick example of this in code.

Here I am with my person object and I ask for its object IDs for the relationship named family. This gives me all my relations and then I can go and fetch these relations in a batch size of 100 and then traverse through these relationships at 100 at a time keeping my memory input rather small and manageable.

Let's move on to NSManaged object context and a new method called refresh all objects.

Refresh all objects does exactly what you expect it to do. It refreshes the objects in your context but preserves unsaved changes and unlike reset on the context your object references remain valid. So you don't have to refetch any references and it is ideal for breaking retain cycles which may have occurred when you traversed a bi directional relationship and looped yourself around.

Also new on NSManaged object context is for those of you using multiple coordinators in your store.

Merged changes from remote context save will take a save notification from one coordinator and apply it to the context in another coordinator. This allows you to have the latest row data in all your context and we'll take care of all the necessary context for you.

In Core Data occasionally you'll run into one particular exception and that sticks out a lot to developers. That's the inability to load a fault.

Why is Core Data unable to load this fault? Well, as I mentioned earlier, Core Data is aggressive about lazily loading objects, you'll only have a portion of your graph in memory and it is possible as we try to traverse a relationship we'll try to have to go back to disk and that object has been deleted out from underneath you.

What's better than throwing in exception, there is a lot of things. We have introduced a new property on the NSManaged object context that allows you to set some basic faulting delegated. Currently should delete inaccessible faults, defaults to yes. If we encounter a fault we'll mark the fault as deleted and any missing attributes will be null, nil or zero. This allows your app to continue on with this object and treat it as a deleted object. No longer will you crash but you'll be able to merely keep going on and show the user what they have expected to see. Now on NSPersistent store coordinator we have two new APIs to introduce.

We introduced these two new APIs because we have seen issues with the way that developers clear up their persistent store.

A few of you have done this. You have gone through and bypassed the Core Data API layer to manipulate your database directly.

Unfortunately this has some unexpected consequences, you may be leaving bad descriptors open and so we have supplied you with destroy persistent store at URL.

Like ad persistent store at URL you take the same options and you can destroy that persistent store and we'll honor all locking protocols as well as clearing out all the related files to the particular store type that you have chosen to use. In that same vein we have introduced replace persistent tore at URL, similar pattern as destroy and if the database at the destination doesn't exist we'll simply copy it into place. One of the problems you have all run into is duplicates.

Having a database with duplicates is not useful. You have written a lot of code to ensure you don't have duplicates.

Core Data can help you out here too.

First let's look at a common pattern you may have used to find duplicates. That's the find and create pattern.

Here as you see, I set up a fetch request and I had to go and look for one particular object to see if it exists before I can create it. If it does exist, I update it.

Well this pattern can be rather racey and it can also lead to more duplicates if I have several threads ingested from multiple data sources. Well Core Data has you covered this year, simply tell us which attributes should be unique across any entity and we'll make sure all instances of that entity keep that unique attribute, be it email addresses, part numbers, UPC, you name it, we'll make sure it is unique across all instances.

Unique instances, unique constraints are best used when your on values that are unmodified after object creation, generally when you create the object these unique constraints should be set once and then never changed to the life of the object.

Changing them could result in conflicts as your unique properties may collide with another object that has the same unique attributes. That's where you can use the recovery methods in the merge policies we talked about previously to address those issues.

Also any of your entities that inherit from a parent that has unique constraints will inherit those unique constraints as well. In this example above you can see the parent has UUID constraint identified as a unique constraint. The sub entity has added email as an addition to its unique constraints.

Now I would like to take a quick demo of showing you how to utilize unique constraints. So here we are, we're using the recipes app we have shown you in previous years and it is available at download off of the developer portal. We've added a new feature, import, down here in the right.

This allows me to import all my favorite recipes that involve apples. Here we go back, you see I added all of my favorite apple recipes however my UI isn't very intuitive and the user may be wondering what's going on when they click. Unfortunately, they have duplicated their data. We can do a lot better here. Let's go back to Xcode and look at our object model and here we are on our entity, I'll select a recipe and now we have a new option over here on the right called unique. This allows me to specify which attributes are unique for this particular entity. In this case we'll have source and external ID.

Thousand when I run the recipe app we'll see I have my original list. I can go, import, and I can select my apple recipes but I'm also impatient, not seeing any UI so I keep clicking. This time we're left with the one single object for all of them, no duplicates, no extra code to find or create, your unique constraints ensure that your uniqueness is there. However, having all of those duplicates isn't ideal.

Getting rid of the duplicates can be a lot of work. That's where Scott Perry is going to come up and show us what we can do about that.

Thanks, Rishi.

So let's say you already have an app and you've got all this duplicated data now, now you have to go and delete them all. Today what you have to do is fetch all of them from memory -- sorry, from the store -- once they're in memory, you mark each of them for deletion and then you have to save down to the persistent store, if you have a lot of objects you're going to have to do that over and over again in order to maintain a low enough memory footprint so that your app stays alive.

It seems kind of silly to just load objects into memory just so you can delete them. This year we have introduced a new API in the form of NSBatch delete request.

NSBatch delete request works like NSBatch update request in that it acts directly in the persistent store without loading any objects into memory.

You can create one using an instance of NSFetchRequest specifying an entity, one or more stores and use predicate or sources or limits to slice up the data in whatever interesting ways you want.

A batch delete request returns a box type NSBatch deletes result and you can configure the request to return a successor failure, the default, the count of the objects that were deleted or the object IDs of the objects in that box. There are a couple of limitations to this. Since none of the objects are loaded in memory, the changes are not reflected into the context and none of your validation rules are run.

Relationships will be deleted out or nullified as appropriate, but that's all the guarantees that you get. There are also no object notifications.

We think this is going to be really helpful for people if they have a lot of duplicates and I would like to show you how it works now. So I have here the same recipes app with a copy of a database I got from my manager. He says one of his kids got ahold of it and added a lot of recipes, like thousands. If we were to go threw this the old way, then we would just fetch all of the objects we want to delete with a fetch request. Then iterate over all of them, deleting them, and then saving the changes with the batch size that we've configured to be 1,000.

If you try doing this you can see in the console here it takes a while.

You can see we're doing -- since we're doing batches in the thousands, we're now in our first batch and it's still going.

This will take a while. We're not going to stand here and wait for it. If we break in a convenient spot we can kill the app and try again using batch deletions. Let's get rid of all this.

Creating a duplicate delete request using the same fetch request that we used before and we're going to choose a counter resultType so that we can see what we have done.

Here we will execute it. It is a lot less code, there's just one single execute request, no looping, no interacting with objects. If we build and run this, -- we'll come back here.

You can see here that in the queries the generator has created a trigger that deletes all of the relationships that need to be cleaned up and we're done.

Back down to a simple number of recipes so that now we can apply the unique restraints.

Scott Perry: That's NSBatch delete request.

Next up I wanted to talk about model versioning.

While we were creating the new version of the recipes app in order to support the import feature we had to add two attributes to the recipe entity source and external ID which Rishi showed you earlier.

While working on this, we open up the model, adding the two attributes build and run and right away we had an error. I highlighted the most important part. We incurred a migration because the model changed but we forgot to include the original source model because it was what we used to change and the pattern of having to copy your old model in order to create a new one is really cumbersome when you're reiterating your apps. And if you forget to deploy a model to the hands of a customer that's running that version it's really dangerous.

This seems to be a case where automatic lightweight migrations should work for you. Now iOS 9 and OS X.11 we have model caching.

Whenever you have a store that's created or migrated or just opened on the new iOS from an older version the managed object model used to create it is cached into the store and it is used by lightweight migrations when they fail to find appropriate source model as sort of a last-ditch effort.

Scott Perry: There are a couple of limitations, this is only available for SQLite stores and the cached model is not available for heavyweight migrations. If you're doing heavyweight migrations you have your model ready anyways because you need to know what you're actually transitioning from and to. Rishi talked earlier about API we added and I wanted to talk now about some changes. For iOS 9 and OS X 10.10 Core Data has adopted all the new language features you have seen in Objective-C running generics and nullability. We have also taken advantage of a new attribute called kind of that allows for easier downcasting. You may not have seen this in other talks but this is really handy for Core Data because normally if you're interacting with an objective type ID you can downcast it to anything, even completely inappropriate types. But using kind of you can attribute type to only be downcast to subtypes of that type. This is going to add a lot of safety to your code from the compiler because it will limit warnings whenever a cache seems like it doesn't make any sense.

Generated subclasses have also been updated to use generics for too many relationships as well as nullability and we have made some other changes to subclass generation as well. In Xcode 6 you would get an implementation file and a header file for using Objective-C containing both the Core Data declarations as well as a place to put all your code and it was thrown over the fence for you to own afterwards. If you change your model, that becomes sort of cumbersome to keep up to date. In Xcode 7 we have added a new file.

This file is an extension or category depending on the language that contains all of the declarations that you're familiar with from the header.

So now the header and implementation file are yours to own and whenever you update your model all you need to do is update this file.

Scott Perry: That's it for changes. I wanted to talk about deprecations, we're getting rid of confinement currency in iOs9 and OS 10.10-- it is marked as deprecated, we're getting rid of it later.

Because the confinement was the default behavior for new managed object context, we have also deprecated in it, and so moving forward you should be using init with con currency type using private cues or main cues for your contexts. If you haven't already moved over to the block API it is really a good idea. Encapsulation makes it a lot easier to reason about your model code and the concurrency debugging is reported as much stronger. I highly recommend checking out the online documentation, the Core Data guides were completely updated this year and Adam Swift also introduced the block API with this really good talk What's New in Core Data on iOS in WWDC 2011.

Last I wanted to talk about performance.

Over time we start adding attributes to the models, the amount of data that's carried by your users gets larger as they hold on to our apps over the years and the ways in which we try to query the data to show them get more interesting, more advanced, and our apps stay fast. But how do you avoid being surprised by performance problems? When you're in development you're dealing with a known data set that may be smaller than what your customers are working with and the simulator is much faster than moving on a device. While that's great for development, the users are using devices with production data.

Luckily we provide tools that allow you to spot patterns that are indicative of performance problems so you can solve them before they become problems in the hands of your clients. I'd like to talk about three things to look out for, the first of which is relationship faults. This is the Core Data Instrument and we have just ran the recipes app and right away in the cache missed Instruments we can see that we blew the cache on three objects we wanted to display. If we look in the middle column we can see their recipe type. And now you remember when we made the recipes app this year we updated it so that the main list view showed the type of the recipe along with the recipe itself but we never changed fetch requests.

We can fix this by adding a relationship key pad for prefetching to the query we used to see our NSFetchResults controller.

Those first -- that first set of cache misses will now no longer be a problem. If we go back to the same Instrument, and look a little bit later in the app, we can see that when we viewed the detail of a recipe we also incurred a number of careers to the database.

This is because the detailed view controller gets its model object from the list and then in the detailed view we display all of the ingredients in that recipe.

We can't use prefetching, because then we would prefetch all of the ingredients for all of the recipes displayed in the list view. In the detailed view, in the controller, we have to execute another fetch request to get those ingredients up into memory. Now we have turned 9 queries into one and we can still use the relationship on the recipe to reverse it and interact with the set it returns because the data is shared on the objects. Lastly, if we look at the fetch Instrument in the Core Data Instruments view we can see that that first fetch request took longer than we would like.

It fetched 85 objects. At the time we only had 85 objects. This is going to be really bad if we have 30,000 like I showed in the demo. The app probably wouldn't even launch. It also it took 15 milliseconds on a Mac Pro, that's going to have a lot of dropped frames on iOS so what we can do to resolve this is add a batch size to our fetch request that's fed into our fetch results controllers so that objects are fetched from the store only as they're needed to display.

The last one I wanted to show is sequence blocking.

If you have really complex fetches that take a lot of time then you can find them by using this argument to your program and it will start printing out data about your fetch requests as they're ran. This case, we have a query and the amount of time it took, which was about, what, almost a tenth of a second and it returned 85 rows. That's pretty slow. We should take a look at doing better.

If we scroll up higher in the console we can see it printed out the file we're using and we can connect to that file using SQLite to figure out what's going on. If we paste in our query after explain query plan then SQ lit will tell us what it is trying to do to fulfill the query with this table. There is a couple of things to note here that we can use as a metrics for how to expect the performance to be. The first is scan table, scan table means that SQLite is going to touch every row, inspect every row to fulfill the query and on the recipe table as we had earlier, that was 30,000 rows and we'll do that twice, so that's not going to be very fast and we have to investigate making it better.

Also, we have use temp B-tree which is a step where SQLite builds its own memory structure out of the data in order to fulfill either sorting or fast searches.

The temp B-tree is being used because of this group by here, if we take a look a little closer, it is because of the source in external ID. We should be able to make this faster by using a compound index. In the Core Data model editor we can add one on the right-hand side here. Now, if we quit SQLite, rebuild our project, perform a migration, and attach to the new database with SQLite we can then see that we're using an index. Using index means that the searches is going to be fast using covering index is even better, it means that the results of that step are in the natural sort order needed for the next step. We have entirely eliminated the temporary B tree but we still have the scan table.

In this case, we're matching duplicate objects. This was the query we used in the demo in order to find objects to delete. It has to scan the entire table, it can't be faster.

The only thing left to do is to make sure that we're off the main thread. In this case we're using a private Q context, but if you were trying to create some composite data do show to a user, you might want to use a nonsequitous fetch request that will get off of the main thread while it's working and then come back when the results are ready.

So those are three common patterns to look out for in your app that allow you to solve those performance problems before they're problems. That's it this year for What's New in Core Data. If you find any problems, please file them. There is a bonus for sample apps code that reproduces right away, it gets fixed first. We're also interested in hearing things that you would like to see in Core Data, feature requests, enhancement ideas, the documentation guides as I said were all revised this year, so if you find any issues with those we would love to know about them as well. For more information, check out the developer portal and our documentation and sample code, you can get support on the dev forums or through DTS. Thank you for coming.

[Applause]