App Thinning in Xcode

ANDERS BERTELRUD: Good morning.

My name is Anders. I would like to welcome you to session 404, App Thinning in Xcode. Now, app thinning has been mentioned on this stage several times this week, but judging from your questions and the number of questions in the lab yesterday, a lot of you are interested and have questions.

So we are going to go into a lot more detail in this session.

In particular we are going to talk about how app thinning affects today's app distribution pipeline. We'll look at how it works today and how it is being improved.

We are going to talk about what you can do in order to make app thinning as efficient as possible. And to make your apps as small as possible. Then we are going to talk about what this means for your work flow as you develop and test your applications and submit them to the App Store. So very quickly let's take a look at how app distribution works today. You use Xcode and build and iOS app.

You test it and submit it to the App Store.

When you submit it, the app is signed with the developer certificate so we know it came from you.

After it has been approved, the App Store re-signs it with the store certificate and hosts it for apps to download.

So along comes a device and downloads your app. What it gets is exactly the same thing as what you uploaded. Let's take a look at what's in that.

First of all, there's your executable code, of course. And there's resources, broadly everything can be categorized into that. Some of your apps are code heavy, have a lot of code and not so many resources and others are the other way around where they are media rich and have some code but it's mostly assets.

We are going to take a look at one of those as an example. Now, with your code you most likely have 64-bit and 32-bit versions. In fact, you may have RMV7. RMV7 has NRV 64 slices in order to run as well as possible on all the hardware that Apple shares.

Of course, your assets, if you have images, you will want to use 1X, 2X, and 3X, Retina, art work to look as great as possible on every type of device. You may further differentiate the art work so that you have some for the iPhone idiom and others for the iPad idiom.

If you have a game or other 3D graphics oriented app, maybe 1X, 2X doesn't mean as much to you as text compression does and number of polygons.

In fact, if you want to take advantage of OpenGL AS and metal, you may have different assets, different textures for that, and you may further differentiate between low quality and high quality so you run as well as possible and look as great as possible on whatever device you have depending on the memory and graphics capability. You probably have audio as well and you may differentiate that too in terms of bit rates.

Of course, there's other data such as the levels of your game. Or if you have a documented application, you might have templates and other information that doesn't make sense to differentiate by device. It is the same for all devices. And many real world apps actually have a bit of everything. So you may have spent a lot of good effort categorizing things into asset catalogs, annotating your assets so the right ones get loaded on the right device.

That's great. That's actually what app slicing takes advantage of.

If you have an ipad Mini, at runtime it will load the RMV7 slice, for the iPad, et cetera. What we are doing with app slicing is making the runtime decision of what assets to load, moving that up into the store.

Right now the content goes onto the device, always is stored on the device, even though it will never be loaded on an iPad 1, so the RM 64 slice won't get used on the ipad Mini.

Therefore, why send it at all? So in the store, it is going to prebuild a version tailored for the ipad Mini. Same thing with an iPhone 6+. In that case, we have 3X art work, RM 64, et cetera.

Similar thing. What is important to note here, you still will be uploading one universal app with all of the different variants of art work and other resources as you have today, and it is in the App Store that the slicing happens.

And what the App Store does actually is to look at all the different device traits of the different devices that your app can support.

Then it looks at the actual content of your app to see how you annotated it with 1X, 2X, et cetera, and it builds premade, separate IPAs that get downloaded. Let's take a look at an example you've seen already, which is demo box, which was shown in the state of the union. In this case, we have an app that is 74-megabytes when built universal. That is for all architectures and for all types of devices.

And by thinning it out, we get between 16 and 29-megabytes, 22 on the average. This is an app that we run the slicing on. There is actually 19 different variants that are produced for all the various combinations of traits. All of that happens invisibly to you or without your effort. That happens just by how you annotated your contents of the app. So that's one level of savings. That's pretty good. Can we do better? Of course, the answer is yes.

There are some things that we want to have all the time on the device. So the code, for one thing. Then there may be some art work, like a story board that is shown when you launch your app or some art work for a main menu, let's say, that you'll always want to have there.

Then there are these other things that yeah, this app will need, but it is not going to need it right now. A typical example that you've heard before is level 19 of a game. The user who is beginning to just play level 1 is not going to need that yet.

There are other examples, too. There could be media rich templates if you have a document rich application. You may want to ship a lot of those. In order to provide a rich experience, you may have a lot of templates, but they might take a lot of space. You may want to load those on-demand. Same with audio for instruments, those kinds of things. There is a good example of tutorial that the user may watch once, and you don't want it to take up space on the device anymore.

If we take a look at one of the slices that we talked about before, it already has been thinned down, sliced so it has one architecture and it has one quality variant of every type of asset.

The next level of partitioning we can do is something that only you know based on the particular domain of your app, which is what you are going to need at any given time, because that is based on the logic in your code.

So in this example, we have images, Metal shaders, those kind of things. We can divide those into those that are shared and always needed that we want to keep in the app itself.

We can have things that come in on demand.

By splitting this up, then you can actually shrink the base footprint of your app and still have access to content.

In this particular case, the asset packs that are built by Xcode based on your asset tags get moved off.

They are stored separately from your IPA in the App Store.

You can download them on-demand.

You can designate some of them as being part of the initial download so when the user downloads the appear app and see the progress bar go to 100 percent, by the time they are ready to startnusing your app, they already have some of the initial content.

You can bring in more content on-demand as needed.

Now, this is not stored in your app's bundle on the device and also not on the container that gets backed up to iCloud. This is stored in memory managed by the system that can cache the on-demand resources from the different applications on the device.

So now in this case if you, in this scenario we actually have space to have more than three levels. We can have more levels than would normally fit if you didn't use on-demand resources.

When we bring in level 4, the system can automatically get rid of the least recentl used asset pack containing your level 1 resources.

Same thing for level 5.

Of course, if the user hasn't used your app for awhile and some other app needs to download on-demand resources, it can reclaim the space. When your app is used again, it will be downloaded one more time. So some things about on-demand resources.

You build asset packs by tagging assets in Xcode. You saw a brief demo of this in the state of the union yesterday. And there will be a whole session on on-demand resources later today. Asset packs can contain any nonexecutable assets.

App slicing works on the asset catalogs in the assets you've put in there. For on-demand resources you can have any loose files and tag them, and they will be part of the asset packs, downloaded from the App Store. They're hosted by the App Store. You submit it all together to the App Store as part of one IPA, but the App Store splits it apart and hosts it separately.

They are downloaded when needed based on your code. They are reclaimed as necessary.

And they are sliced just like all the other content.

So you can put asset packs in there. In fact, if you tag assets that are in asset packs, Xcode will automatically build them for you in asset catalogs. Let's take a look at the size savings -- oh, right, there will be a session this afternoon which will be in Pacific Heights at 4:30.

Let's take a look at what kind of slice savings we get then. After slicing, we had on average 22 megabytes for one of the slices.

In this case, we get down to between 11 and 18-megabytes depending on the device for the core application and the assets themselves are between 5 and 11. The point is that not all of those are on the device at the same time.

So the sum total is about eight megabytes in this case. This is a fairly small app that you can see that for a large application especially one with lots of media, the savings really add up.

Some obvious advantages of making your app smaller is a better user experience. There are affordable devices that have eight gigabytes of storage, and you really want to make the most use of that. You don't want anything on the device that isn't actually going to be needed.

By using app thinning, you can support more of those devices.

Also this means shorter download times, less to send over the wire which is great.

That makes it easier for you to stay under the 100-megabyte over the air size limit, the cellular limit.

What this means really for you is that you can support more types of devices with less compromise. If you have been thinking about I really want to use Metal and run great on high powered devices but I also want to be able to use the less powerful devices, have customers run well on those, what do I do? This frees you from some of that compromise. You can now submit, it matters a lot less how much you submit to the store as it does what each user gets from the store.

What this means for you in the longer term is you can actually now use some of that reclaimed space to add those features that you couldn't fit in because of size. This doesn't mean you make it huge again, but it means it is less compromise for you when you look at your size versus the things you want to put into your app. So after that overview, I would like to ask Patrick, my colleague Patrick Heynen to come in and give you some of the details on how to do that. PATRICK HEYNEN: Thank you, Anders. Hi. I'm Patrick Heynen. I would like to talk a little bit more in detail about asset slicing. How does it work? Well, I'm happy to say that asset slicing and app thinning have been seamlessly integrated into all the build, publish and export and workflows of Xcode that you may already be familiar with in building your applications.

All that really happens to actually make you realize asset slicing and app thinning is that we do a post process of your build products, of your asset catalog and executable files to tailor the variants for all the different devices that you are trying to build for.

So what do you need to do? Well, the good news is that you are probably doing it already, or doing it already.

If you have been developing iOS apps for a while, you have already been incorporating 1X, 2X, and 3X art work to capture the range of displays on the embedded devices and may even be using asset catalogs which have been around since iOS 7.

That's really all you need to have.

What are asset catalogs and what role do they play? Asset catalogues are a facility to organize your resources according to the relevant device traits of those resources that they are important and pertinent for.

There's an important note here.

In order to take advantage of app thinning, your resources must be contained within asset catalogs. We do not thin loose resources.

This is not as strong a constraint as you might imagine. I'll get to that in a little moment. It is an important note.

So I mentioned device traits. So in the asset catalog world, devices have a key set of characteristics for which assets can be optimized for. These include things that you are familiar with, screen resolutions, 2X and 3X and also device family. Whether it is an iPad or an iPhone device idiom.

New to iOS 9 in Xcode 7, we are also exposing a new set of device traits that are based on the hardware characteristics of our devices.

In particular, graphics capabilities and memory levels.

This really arises out of two needs. One, the range of devices we support now have a huge range of performance characteristics, different between the lowest end device and highest end device. It is not the case that a single resource is appropriate for all of those classes of devices. It becomes really hard to make the right compromise in terms of what your end user experience is going to be. The same goes for memory levels between the low end and high end devices. The new device traits allow you to tailor assets to those different capabilities. What kind of content goes into an asset catalog? Well, the most popular by far is the one that we debuted asset catalogs with, and that's named images.

This basically is your art work resources for your application in typically bitmap resources or vectors, in the png, jpg or pdf format that you integrate into your project. These get compiled into an optimized delivery format using advanced compression codex with really good space saving characteristics and delivered to your application through the UI kit APIs. New to iOS 9 and Xcode 7 is a new data class, at least one new data class.

That is name data.

What is this for? Name data allows you to store arbitrary file content.

Now, this in fact alleviates some of the concern you might have had at my statement that we only thin asset catalogue based content. You ask: What if I'm trying to thin non-image based resources? That is what name data is for. It allows you to put any arbitrary file content stored in your asset catalog and classified according to those hardware capabilities I mentioned to you earlier.

And then at runtime you use the new NS DataAsset class in the UI kit in OS X, by the way, to retrieve this content into your application.

Brief side bar here for a moment. The asset catalog features I'm describing, the name data and some of the ones coming up, are available in OS X as well. We don't thin applications for OS X, but since at the catalog store our source artifact formats are intended to capture your sources across all your targets, including possibly OS X and Watch. All these features work there. The APIs are present on all the platforms. Just the thinning is particular to iOS.

Side bar end.

There's another new exciting data class that we introduced in Xcode 7 and iOS 9 and that's Sprite atlases.

What this represents is an exciting combination of asset catalogs and SpriteKit, allowing you to organize your image assets in the normal way in asset catalogs, and group them, name them into a Sprite atlas, and what this does is automatically creates texture atlases at build time that you can retrieve through the SK texture atlas class from those image assets.

The key thing here is, if you have done all the usual organization on iPhone, iPad, it will automatically create a thinned version of those compiled texture atlases and treat those as thinnable resources that will be thinned automatically by the App Store and by the other workflows I'm going to talk about in a moment. So how do we get from asset catalogs to app thinning? Well, it is really quite simple and you may have guessed it already.

Every asset in the asset catalog has trademark up information that describes what characteristics that resource is pertinent and useful for.

At build time when we do the post processing, these traits are used to route those resources to the relevant thinned app variants. It's as simple as that. I would like to point out an important note. Asset catalogs have been around for awhile. They have always given you the ability at runtime to select the right resource appropriate at runtime to the device that's requesting that resource.

It is this very same algorithm and the very same selection criteria used to do the routing of the rich thinned app variants. If your application is behaving correctly today on a range of devices, odds are it will work well with app thinning, because it is reallythe same mechanism.

Okay. I mentioned the word organization in conjunction a little while ago. I would like to emphasize that for a moment. Cataloging efficiently in asset catalogs is absolutely key.

Why is that? Because a robust mark up of your resources means that there is less redundancy in the slice application variants that are produced.

You don't have extra payload being produced in those app variants that are never going to be used at runtime on the target device that app variant is built for.

For example, even though it might have worked perfectly before, it may not make sense to leave an asset as universal if they are only used on one device family. Consider, for example, some button art work for a pop over UI for your iPad version of your app, but that never gets presented on the iPhone versions of the application. It would have worked fine before to have it as a universal asset and drop it in there. However, it may now be a good idea to revisit and categorize that appropriately so you don't have it delivered to the iPhone versions of your apps where it doesn't make much sense. Okay. So those are the basics. Those are the theories of how this system works. Let's get practical and talk about some real world workflows and how this impacts them.

Let's talk first about creation. How do you create asset catalogs? The predominant way in which you create asset catalog content is through the asset catalog editor in Xcode.

This is easy to use user interface in the IDE. You may be familiar with it. All it takes to create a new asset is to add a new item, and there's a range you can see here. Here we see the data sets, image sets, Sprite atlases as well as the other data types for other platforms. When you add this to your project, you are presented with an organization array and spectrum on the left that shows you the device traits that you can catalogue under, and expose how varied and diverse you want to provide resources for.

You drop your assets into the appropriate wells, and you are off to the races, and you're done. So that's great. That's really easy to use.

What if your team cannot use Xcode for asset production? What if you are a game studio that has a heavy investment in an existing asset tool chain or pipeline where it is not practical to put Xcode in front of the content engineers or the designers that are producing and are needing to attribute the art work appropriately? I'm happy to say that we consider this, and we carefully designed the asset catalog feature in conjunction with thinning to accommodate these scenarios.

Specifically, we've designed it in a way where it should be very easy to export image sets and data sets from existing asset pipelines.

How are we going to accomplish this? We are going to do this through the XC asset source artifact format.

I'm happy to say that we are going to be documenting and publishing to allow external tool chains to implement this format easily.

Now, it really is not much of a format because it is a simple folder structure in JSON markup and really easy to work with.

I want to emphasize, this is not some kind of import format that you sort of create and then you import into Xcode and then it becomes a project artifact. This is literally the source artifact format that the Xcode build system works with natively at build time to process and produce your final runtime artifacts.

Let's take a look at what this means in detail.

We take the previous example of the AirPlay icon art work and look. This is what it looks like on disk. This is in fact the format.

Very simple.

All it consists of is a folder naming convention, consisting of the name of the asset, this is the name you retrieve it from in code with UI image names, and then the individual art work resources contained within that folder.

And I want to point out that even though this particular example uses a pretty well-known semi-canonical file naming convention for showing scale factor and device idiom, there is no requirement to have any specific file naming convention. You can use whatever is natural to your workflow or tool chain or whatever you like.

Why is that? Because there's this all important third element to this design, which is the contents.JSON file.

The contents.JSON file is the file that contains all the asset markup information and joins everything together and tells the system what traits are relevant for a particular resource.

So if we were to actually look at the AirPlay icon example we showed in the Xcode editor before, this is what it looks like. As you can see, it is straightforward. It is just an array of the individual images. You can see the individual file names and then the device idiom and scale factor markup information.

These properties, these key value pairs that you see here, these are the things that will be documented in the SES source artifact format.

Taking the example one level further, name data, what we call data sets, look unsurprisingly similar.

Again, the file names are for the individual resources are there and then the attribution of universal type identifier and memory or graphics future set class.

And you can see it is really quite easy to work with. You can see from the way this works, how they join the file names with the properties that this is why it doesn't matter what file name you put into those data sets and image sets. What is important is that you get this contents.JSON correct.

What does this allow you to do? What is this for? I would like to work through a hypothetical example here of image set creation with Xcode. The hypothetical workflow I'm going to show here is using the PhotoShop CC generator in PhotoShop.

So let's say your designer created beautiful art work and they decided iPads are going to be blue today and iPhones will be red. That's just the way we roll.

Designers do that sometimes. They have the beautiful master file and they set out all these variants here. They actually set their master file up with a generator workflow, turned on image asset generation, and have their layer list constructed with the appropriate naming coventions and default layer to export directly into an image set, and they have a special hacked version of the generator that they carefully extended, this is the hypothetical piece of my example, to produce the contents.JSON. Here is what the output would be from the PhotoShop workflow. For those who used generator before, the photo layout will be familiar. There is the assets folder. Inside is the image set that was generated, all of the different art work entities and the contents.JSON. All of that is required to make this work is drag that image set over and drop it into the folder, underneath the XE assets folder and you're off to the races.

That's it. A few words about this kind of dynamic integration with your Xcode project.

The only requirement is that your project must have an XE asset folder. That's really the only requirement from a project file level, from an Xcode project file level.

Anything contained within that can have an arbitrary hierarchy, arbitrary structure.

It has to conform to the image set and data set, you can nest that arbitrarily deep and you can use your own file system organization underneath that. At build time, Xcode will recursively descend that hierarchy, discover all the image sets and data sets, the spread atlases and automatically build them, providing that you authored the image sets and data sets correctly.

Now we know how to create asset catalogs.

Let's move to the next step, building. So perhaps the most common and most important build workflow is just when you are sitting there working on your app you have a tether device or you are working in a simulator, working on a feature. How does thinning apply in that situation? I'm happy to say that the build and run workflow automatically thins resources for the currently active run destination in Xcode.

So what this means is that every time you hit build, it is actually only going to consider and produce and automatically create the proper runtime asset catalog for the target device that you are using.

And this is supported for all simulator and device run destinations.

So it is a really great way to immediately take advantage of thinning.

This functionality is also triggered by or controlled by a new build setting that has been exposed in the asset catalog sections of the target editor. It is called enable building only active resources. This is handy. What if you're trying to figure out if a problem is being introduced by thinning or what the impact of thinning is overall on a particular target device. You can toggle this on and off for that purpose. Some other benefits of this build workflow is that it dramatically can speed up iterative or incremental builds especially if your application is very content heavy. It really only processes and works with the resources that are appropriate for that target device at the time. It doesn't have to go through everything like it used to before when it was building universal, a universal version of your application.

Another key benefit is this allows you to easily test the impact of cataloging changes on the thinned outputs for a particular device or set of devices. If you are trying to understand the impact of the sizing of your overall built application or other aspects of your user experience, you can make those cataloging changes quickly, build and run for that device and then check other devices by switching between the other devices and the run destination in Xcode and test those changes fairly quickly and easily. Now we've covered build.

What's next? Well, distribution. This is the essential step involved with thinned applications.

Now, there are many ways that we allow you to distribute your iOS applications.

Enable you to distribute your iOS applications. The biggest and most significant one, of course, is the one that your end users, the customers are using through the App Store purchase. This is the mechanism Anders alluded to where you supply your universal IPA.

The store does the work, processes all the thinned variants. When the user is purchasing the app, it's going to automatically select the right device variant from the store, and download and installs it. You're done, fully automatic.

Nothing to worry about.

Before you release the application, you may want to make that same exact experience available to your beta C testers.

We will be supporting test distribution through -- thinned distribution through TestFlight as well. Then, of course, there is ad hoc and enterprise distribution which is a very common way for you to distribute applications in your organization or to your team for QA testing or that kind of thing.

That's an incredibly important way of distribution. We have built thinning support into that as well. Finally, there's the Xcode service and ECos server.

That has been upgraded to support distributing thinned applications as well. I'll go into some details in a moment. All of the methods I outlined are fairly automatic. You don't have to really do much. Ad hoc enterprise solution is something I want to focus on for a moment. That is the one where you will be hosting this distribution flow yourself and you may need to understand some of the pieces of how this is put together and how it works.

Let's talk about that. How does that work? Well, one functionality we've added to Xcode is that you can now export for a specific device from the organizer.

This is great if you are trying to test a build or hand off a build to somebody with a specific device. You can immediately create an IPA for that specific device directly from the organizer, correctly provisioned for that device. Coming in a future seed we will be providing you a way to export all of the thinned variants for all of the compatible devices giving you one complete distribution unit for all the possible supportive devices of your application for a given release. This is important. You may not know or be able to generate all of the thinned variants for all the devices. You want to get the full set. What this will do is actually process that universal app build archive that you have, produce all of the thinned variants that are appropriate for all of the different compatible devices and put them into one set of products.

But now you've got a big bushel full of applications.

You may ask yourself how do I get the right application to the right device? That's kind of a big problem now.

Well, the most common way that apps are delivered to devices with enterprise and ad hoc, of course, is over the air installation. I would like to go through that and how that works with thinning for a moment. The first step is to save for ad hoc or enterprise distribution. It is important here that you include the distribution manifest option, the check box on the lower left there.

What does this do? Well, what this does is when Xcode is generating that exported set of IPAs, it is going to generate a manifest list containing URLs for each of the app variants that it produces.

What's more, it is going to index all of these URLs by the supported product type of that app variant.

The impact of this is that when a device tries to pull that, tries to install one of the apps, using the over the air manifest PO list, it will walk through the list, find the appropriate URL for its device, product type, and install it and pull that one down. That's how it works. This is a mechanism conceptually very similar to what actually happens with the store with your thinned application, but all of these moving pieces are all hosted by you on your Web server and through the mechanisms produced by Xcode. What if we take it one step even further? That's what the Xcode server continuous integration service does.

What Xcode server does is it turns all of this mechanism I just described into one turnkey solution for building, hosting and distributing thinned apps, including hosting on-demand asset packs.

It is quite simple. The Xcode server, you set up your integration box in Xcode in conjunction with your Xcode server deployment. The bot builds your application automatically, produces all of the variants, produces the manifest P list and even wraps a nice Web service around it where you point the device at the Xcode server website and select a particular build, all you need to do is click on the install button and all of the mechanism I previously described of referencing through the manifest P list, pulling the appropriate variant, it all happens automatically.

It is simple as pointing your device at the to the Web server portal and going.

I encourage you if you want to get a feel for how everything moves around, when it becomes available, I encourage you to look at this. This is the same pattern you might use to build your own enterprise distribution or ad hoc distribution workflow where you host things yourself.

None of these pieces are things that you couldn't build yourself.

The Xcode service, to wrap that piece up, is a streamlined installation method for thinned app variants using the over the air manifest.

Nice, simple solution, one stop. Okay. So now we've talked about how to create asset catalogs to enable app slicing. We talked about how that integrates with your build workflows and we talked about the different methods for distribution and how app thinning interacts with those.

With that I will hand it back over to Anders for the conclusion.

ANDERS BERTELRUD: Thank you, Patrick.

All right. So in this session we have seen quickly how the app distribution workflow works today, how app thinning is improving on that; the ways in which you can help this be more efficient by tagging your assets properly and by using some of the distribution features we have.

And what this means for your workflow in terms of testing and then submitting to the App Store.

So a bit of a call to action here.

What you should do, keep creating tailored versions of assets. That's great because you want your app to look its very best and run its very best on every kind of device. With app thinning we are freeing you a little bit from the size restriction of uploading now.

Every user is going to get only what they need.

Keep creating tailored versions of assets. Use the asset catalogs to organize your assets. You can now also put data and Sprite atlases in there and have those tailored for the device as Patrick mentioned.

Also test your thinned app variants using Xcode. This will become important now because we are only delivering the bits that every device needs. You can do this using Xcode's destination as Patrick mentioned and using Xcode server to build the various thin variants and testing them on real devices.

And take advantage of on-demand resources to tag your assets and to have those built into asset packs for later on-demand loading.

I want to call out a couple of places to go for more information here. One is the app thinning guide, the chapter in the app distribution guide that you will find on the developer website at Apple, and the other is the asset catalogue format documentation that's going to be put up on the Apple developer website. We have related sessions. The first couple of ones have already been, but you can catch them on video.

There is another one, What's New In SpriteKit that is going to talk about this some more starting in 20 minutes in Mission. It focuses on the on-demand resources part in Pacific Heights at 4:30 today, so I encourage you all to check that out.

And DemoBots will be talked about in greater detail in Mission tomorrow at 1:30.

We have a couple of labs. Come down and see us in the Labs to learn more about this and have a great rest of the WWDC.

[Applause]