Explore Packages and Projects with Xcode Playgrounds
Xcode Playgrounds helps developers explore Swift and framework APIs and provides a scratchpad for rapid experimentation. Learn how Xcode Playgrounds utilizes Xcode's modern build system, provides improved support for resources, and integrates into your projects, frameworks, and Swift packages to improve your documentation and development workflow.
Chris Miles: Welcome to Explore Packages and Projects with Xcode Playgrounds.
I'm Chris Miles, an Engineering Manager on the Xcode team.
Developers love using Xcode Playgrounds to quickly try out new ideas and to learn and explore APIs and frameworks.
Today I'm excited to tell you about enhancements we've made to Xcode 12 to make Playgrounds work seamlessly with Swift packages and frameworks in your projects.
In Xcode 12, Playgrounds is fully integrated with Xcode's modern build system.
That integration brings with it a lot of improvements to help Playgrounds work with packages, projects, and resources.
First, I'll show you how Playgrounds now works well in Swift packages and makes a great option for package documentation containing runnable code.
Then I'm going to demonstrate improvements for Playground's importing project targets like frameworks and package dependencies.
In the final part of the presentation, I'm going to demonstrate working with resources that need build support like ML models and asset catalogs in a Playground.
So let's get started by looking at how Xcode Playgrounds works with Swift Packages.
Here I have a Swift Package checked out called Nutrition Facts.
I'm going to open it in Xcode by double clicking Package.swift.
Nutrition Facts provides accurate nutritional information for many common types of food.
It's a really handy package to use in recipe and food apps.
What's great about Nutrition Facts is that the authors have included rich documentation explaining how to use the API, and even better, the documentation was written as a Playground.
Playgrounds are great for documentation and they now work seamlessly with Packages in Xcode 12.
So let's take a look.
In the navigator I'm going to open the Playgrounds folder and select the Nutrition Facts Playground.
Here we see a description of the Package followed by a detailed documentation about how to use the API along with example code snippets.
And, being a Playground, we can run that code.
Let's do that now by hitting execute in the toolbar.
The Playground will execute.
And on the right, we see a live view.
That is a view provided by the package.
It shows Nutrition Facts in a familiar format.
In the editor we see an example of looking up an item by identifier and fetching nutritional information about it.
The Playground helps us understand the code by showing the result of each statement in the results bar on the right.
As you can see, Playgrounds in Xcode 12 works seamlessly with packages.
This was made possible because of Playground's integration with the modern build system which knows how to build packages and make them available for importing into Playground code.
I recommend making use of a Playground the next time you need to write documentation or tutorial for your Swift package.
Don't forget to take advantage of the fact that users can run the example code and see live results in the Playground.
Next, I'd like to show you how Xcode 12 makes it easier for Playground's to use framework and package dependencies from a project.
I'm going to use a project called Fruta.
And let's open it in Xcode and run the project in the simulator.
Fruta is an iOS and Mac App for browsing and ordering smoothies.
You can even purchase smoothie recipes with Apple Pay so you can make your own smoothies at home.
Let's take a look at the app in the simulator.
We see a list of some of the smoothies available, and if we select one, we can see an option to buy it with Apple Pay.
And, scrolling up, we see some of the ingredients that are in the smoothie.
Let's select one to see a detailed image and tap the "i" button to see nutritional facts about that ingredient.
Now, you may recognize this view as the view provided by the Nutrition Facts package.
This project has a dependency on that package, as you can see in the left in the navigator.
Even as a project dependency, any Playground's in that package will still be available.
Let's take a look by opening the package and opening the Nutrition Facts Playground that we opened earlier.
Here we see the same content and, like before, we can execute the Playground to see the results.
So Playgrounds in packages are available whether you open the package directly, like we did at the start, or whether it's part of your project as a package dependency, like it is here.
Just remember, that like all files in a package dependency, the Playground will be read-only.
This project also contains a framework called Utility Views.
Let's take a look at the project editor where we can see the utility views framework target.
It is now easier than before to use your frameworks in Playgrounds as long as they are in the same workspace.
Let me show you an example.
In the navigator I'm going to open the Playground's folder and we see a Playground called Smoothie Lab.
Let's open Smoothie Lab.
Notice some build activity at the top.
This was due to a new Playground option in Xcode 12 called Build Active Scheme.
Let me show you by opening the inspector.
And here we see in the Playground settings the new option at the bottom of the settings.
When Build Active Scheme is enabled, it tells Xcode to automatically build all targets in the currently selected scheme and make any modules importable to the Playground.
So that's why when we opened the Playground, Xcode built all of the targets for the currently selected scheme and the Playground could import the utility views framework and the Nutrition Facts package.
Smoothie Lab is a utility Playground that we use to design new smoothies.
We can select some raw ingredients, combine them in various quantities, and then inspect their nutritional values.
We can even chart a calorie breakdown using a chart view from the utility views framework.
Let's run the Playground to see the calorie breakdown for this smoothie.
In the live view, we see a chart of the calorie breakdown.
Now, we usually like to keep the fat content to be less than about 25 percent.
So let's tap the chart to take a look at the percentages.
That's about 33 percent in this case.
So perhaps we could lower the amount of peanut butter in this smoothie, but I'll work on that later.
As you can see, this Playground is a handy tool for helping us to combine ingredients to design well-balanced smoothies, at least from a nutritional perspective.
We haven't yet designed an algorithm for calculating how good a smoothie might taste, but I'm fine with that, as tasting as my favorite part of smoothie design.
Playgrounds can now work seamlessly with project targets such as frameworks and packages.
Xcode will take care of building those targets so that they can be imported into your Playground code.
Just make sure to enable Build Active Scheme.
The good news is that this option is enabled by default for new Playgrounds.
Also make sure that the module you want to import is either part of the active scheme or a dependency on another target built by that scheme.
Another great enhancement in Xcode 12 is the availability of full build logs for Playgrounds.
Let's take a look at the build log for Smoothie Lab.
We'll switch to the report navigator.
And at the top we see the build log for this Smoothie Lab Playground.
So let's select it to open the build logs.
In the build logs we see all of the build details for the targets that were built as part of the active scheme, as well as the module that was built to support the Playground containing compiled sources and resources from within the Playground.
Build logs are invaluable for diagnosing build issues and we're happy that they are now available for Playgrounds.
Another benefit of Playground's integration with Xcode modern build system is that all resource types supported by the build system are now supported in Playgrounds.
Let me show you an example of using some resources in a Playground that require build system support.
Returning to the Fruta project, I'd like to add a feature to Fruta where the user could point their iPhone's camera at their favorite fruit and Fruta would magically suggest smoothies that match the fruit that it could see.
To do this, I'd like to use machine learning to perform object detection and image classification.
There are a number of classifiers available so I'd like to try one of them out with a few sample images to see if it'll be suitable for the task.
And I'm going to do that using a Playground in Xcode.
First, let's get some sample fruit images to test with.
And I think we happen to have some already available in the Fruta project.
Let's switch back to the simulator and navigate back.
And remember we saw the image of some oranges before.
And if we dismiss that we see images of the other ingredients.
So let's find these in the project.
I'm going to hide the simulator as we don't need that anymore.
Expand the shared folder.
And let's take a look in the asset catalog.
We see an App icon, some color sets, and, in the ingredients folder, we see all the images representing the ingredients that we put in our smoothies.
I think this will be a good sample set for testing our machine learning model.
So let's make a copy of this by option-dragging this out to the desktop.
And now we don't need the project open so we'll close that and instead create a new Playground with File > New Playground.
We'll use an iOS blank Playground.
And let's call it "MLFruta" and store it on the desktop.
I'm going to expand the size of the window to give us some more working space.
And now let's drag in the asset catalog into the resources folder of the Playground.
Notice that Playground compiles the asset catalog for us so that we can access the resources from our code.
We can do that using UIImageNamed, specify the name of the folder within the asset catalog, and the name of an image.
And now let's execute the Playground to see the results.
In the results we see the width of the image.
And if we're quick look the image, we see a preview.
That looks great.
So we have access to images from that asset catalog.
Now we just need a machine learning model.
I'm going to switch to Safari where I've been browsing the machine learning section of the Apple Developer Website.
In the CoreML Models page we see a number of image classification and object detection models.
I'd like to try out this model called YOLOv3, which is able to detect multiple objects in a scene and classify each of them.
I think that would be good for pointing the camera at a bowl of fruit and having it identify all the fruit that it could see in the bowl.
So let's click View Models and we can download the full precision model at the top.
To save us time, I've already done this.
So let's minimize Safari.
And I'll drag in the model from the desktop into the resources section of the Playground.
Xcode will automatically compile a model for us.
And while it's doing that, let's select the model to see details about it in the editor.
An important detail is the model class up near the top here.
It tells us the name of the class that will be automatically generated by the build system for use in our Swift code.
So let's make use of that now.
Returning to the Playground, let's add the code to import CoreML.
Then I'll call my variable "yoloModel" and we'll use the class that was automatically generated for us, YOLOv3, to create a new instance with a default configuration.
And we'll fetch the model property.
Let's run the Playground to see the results.
We get back a model description, which we can preview here and see details about the inputs outputs and other properties.
So that's great.
We now have a machine learning model and some images to test with.
So we just need to combine them together.
An easy way to do that is using the Vision framework.
I'm going to replace this code and instead import the Vision framework.
I'll define an array with three ingredients to start testing with.
And then we can use our YOLO model to create an instance of the Vision framework's CoreML model.
And with that, we can create a CoreML request that we can use to make machine learning requests on images.
Now, I'd like to visualize the results of these requests by showing each image and drawing a rectangle around any objects detected in the image.
To help do that, I've got some code that I've used in the past.
Let me Quick Look it for you.
It defines a couple of structs to hold the results of our object detection and some SwiftUI views that we use to show the images and draw rectangles around any objects detected in the image.
We won't go into detail about this code, but we'll make it available on our session website.
So after dragging the support code into the sources section of the Playground, Xcode will compile that.
And now we can make use of the code.
We'll iterate over the ingredient names, fetch an image for each one from the asset catalog, and then we'll create an image request handler with that image.
We can use that to perform our machine learning request.
Then we just need to pass the results.
Each result is a recognized object observation.
We'll enumerate the observations, we'll fetch the label with the highest confidence, and the bounding box of that object within the image and return that as a recognized object.
Finally, for each image, we'll return the results as an object detection result.
And let's preview the results that we get at the bottom.
Xcode will iterate over the three images and then we can preview the results.
Indeed, we have a result for each image so let's visualize them.
We can do that by setting a Playground Live View to the recognized object visualizer and pass it the results.
Let's continue running.
On the right we say the results of our object detection.
The first image we passed in was of a banana and the orange indicates that a banana was detected with 100 percent accuracy.
So that's a pretty good result.
The second image was of three oranges and we can see that all three oranges were detected with high confidence.
So that worked really well.
The third image was of a bottle of almond milk and we can see that the machine learning model detected a bottle with pretty high confidence, although it didn't classify it specifically as a bottle of milk or almond milk.
And that would be because this model has been trained to detect bottles but hasn't been specifically trying to detect bottles of milk or bottles of almond milk.
And I think that's fine.
For what this model's been trained for, it performed really well.
So I think this would be the model to use for our app.
The next step would be to go ahead and train that model even further with the types of ingredients that we want to detect for our use case.
To find out more about how to train your ML models, I suggest watching the WWDC 2019 session called, "Training Object Detection Models in CreateML." So that was an example of using resources that require build system support in Playgrounds.
Let's briefly summarize what I've covered in this presentation.
In Xcode 12, Playgrounds is fully integrated with the modern build system.
When the new option, Build Active Scheme is enabled, all targets in the currently selected scheme will be automatically built and made importable into Playgrounds that are in the same workspace.
Any frameworks or Swift Packages can then be imported into your Playground code.
All resource types that the build system knows about can now be used in Playgrounds.
For example, asset catalogues and ML models, like I demonstrated.
Full Playground build logs are now available in the Report Navigator, making it much easier to diagnose Playground build issues.
Playgrounds are great for documenting packages and frameworks with runnable code.
And don't forget that Playgrounds are a handy tool for quickly coding up new ideas or writing utility code.
So I look forward to hearing how Playgrounds are able to help you make your packages and projects even better.
Thank you for watching.
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.