Understanding Images in Vision Framework

Good morning. My name is Brittany Weinert, and I'm a software engineer on the Vision Framework Team. This year the Vision Team has a lot of exciting new updates that we think you're all going to love. Because we have so much new stuff to cover, we're going to dive right into the new features. If you're completely new to Vision, don't worry. You should still be able to follow along, and our hope is that the new capabilities that we introduce today will motivate you to learn about Vision and to use it in your apps. Today well be covering four completely new topics, saliency, image classification, Image Similarity, and face quality. We also have some technology upgrades for the Object Tracker and Face Landmarks as well as new detectors and improved Core ML support. Today, I'm going to be talking about saliency. Let's start with a definition. I'm about to show you a photo, and I want you to pay attention to where your eyes are first drawn. When you first saw this photo of the three puffins sitting on a cliff, did you notice what stood out to you first? According to our models, most of you looked at the puffins faces first.

This is saliency. There are two types of saliency, attention based and objectness based. The overlay that you saw on the puffin image just now called the heatmap was generated by attention based saliency. But before we get into more visual examples, I want to go over the basics of each algorithm.

Attention based saliency is a human aspected saliency, and by this, I mean that the attention based saliency models were generated by where people looked when they were shown a series of images.

This means that the heatmap reflects and highlights where people first look when they're shown an image.

Objectness based saliency on the other hand was trained on subject segmentation in an image with the goal to highlight the foreground objects or the subjects of an image. So, in the heatmap, the subjects or foreground objects should be highlighted. Let's look at some examples now. So, here are the puffins from earlier. Here's the attention based heatmap overlaid on the image, and here's the objectness based heatmap. As I said, people tend to look at the puffins' faces first, so the area around the puffins' heads is very salient for the attention based heatmap.

For objectness, we're just trying to pick up the subjects, and in this case, it's the three puffins. So, all the puffins are highlighted.

Let's look at how saliency works with images of people. For attention based saliency, the areas around peoples' faces tend to be the most salient, unsurprisingly because we tend to look at people's faces first. For objectness based saliency, if the person is the subject of the image, the entire person should be highlighted.

So, attention based saliency though is the more complicated of the two saliencies, I'd say, because it is determined by a number of very human factors.

And the number, the main factors that determine attention based saliency and what's salient or not, is contrast, faces, subjects, horizons, and light.

But interestingly enough, it can also be affected by perceived motion. In this example, the umbrella colors really pop, so the area around the umbrella is salient, but the road is also salient because our eyes try to track where the umbrella is headed.

For objectness based saliency, we just pick up on the umbrella guy. So, I could do this all day and show you more examples, but honestly, the best way to understand saliency is to try it out for yourself. I encourage everybody to download the Saliency app and try it on their own photo libraries. So, let's get into what's returned from the saliency request, mainly the heatmap. So, the images that I've been showing to you up until now, the heatmap has been scaled, overlaid, and colorized and put onto the image, but in actuality, the heatmap is a very small CV pixel buffer that's made up of Floats in the range of 0 to 1, 0 designating nonsalient and 1 being most salient. And there's extra code that you'd have to do to get the exact same effect like you see here. But let's go into how to formulate a request at the very basic level.

Okay. So, first we start out with a VNImageRequestHandler to handle a single image.

Next, you choose the algorithm that you want to run, in this case, AttentionBasedSaliency, and set the revision if you always want to be using the same algorithm.

Next, you call perform request, like you usually would, and if it's successful, the results property on the request should be populated with a VNSaliencyImageObservation.

To access the heatmap, you call the pixelBuffer property on the VNSaliencyImageObservation like so. If you wanted to do objectness based saliency, all you would have to do is change the request name and the revision to be objectness.

So, for attention, it's VNGenerateAttentionBased SaliencyImageRequest and for objectness, it's VNGenerateObjectnessBased SaliencyRequest. So, let's get into another tool other than at heatmap, the bounding box. The bounding boxes encapsulate all the salient regions in an image. For attention based saliency, you should always have one bounding box, and for objectness based saliency, you can have up to three bounding boxes.

The bounding boxes are in normalized coordinate space with respect to the image, the original image, and the lower left-hand corner is the origin point, much like bounding boxes returned by other algorithms in Vision. So, I wrote up a small method to show how to access the bounding boxes and use them. Here we have a VNSaliencyImageObservation, and all you have to do is access the salientObjects property on that observation, and you should get a list of bounding boxes, and you can access them like so. Okay. So, now that you know how to formulate a request and now that you know what saliency is, let's get into some of the use cases. First, for a bit of fun, you can use saliency as a graphical mask to edit your photos with. So, here you have the heatmaps. On the left-hand side, I've desaturated all the nonsalient regions, and on the right-hand side, I've added a Gaussian blur to all the nonsalient regions. It really makes the subjects pop.

Another use case of saliency is you can enhance your photo viewing experience.

So, let's say that you're at home. You're sitting on the couch, and either your TV or your computer has gone into standby mode, and it's going through your photo library. A lot of times, these photo-showing algorithms can be a little bit awkward. They zoom into seemingly random parts of the image, and it's not always what you expect. But with saliency, you always know where the subjects are, so you can get a more documentary-like effect like this. Finally, saliency works really great with other vision algorithms.

Let's say we have an image, and we want to classify the objects in the image.

We can run objectness based saliency to pick up on the objects in the image, crop the image to the bounding boxes returned by objectness based saliency, and run these crops through the algorithm through a image classification algorithm to find out what the objects are. So, not only do you know where they are in the image because of the bounding boxes, but it allows you the hone in on what the objects are by just picking out the crops that have those objects in it. Now, you can already classify things with Core ML, but this year, Vision has new image classification technique that Rohan will now present to you.

Good morning. My name is Rohan Chandra, and I'm a researcher on the Vision Team. Today, I'm going to be talking about some of the new image classification requests we're introducing to the Vision API this year.

Now, image classification as a task is fundamentally meant to answer the question, what are the objects that appear in my image.

Many of you will already be familiar with image classification. You may have used Create ML or Core ML to train your own classification networks on your own data as we showed in the Vision with Core ML talk last year.

Others of you may have been interested in image classification but felt you lacked the resources or the expertise to develop your own networks. In practice, developing a large-scale classification network from scratch can take millions of images to annotate, thousands of hours to train, and very specialized domain expertise to develop.

We here at Apple have already gone through this process, and so we wanted to share our large-scale, on-device classification network with you so that you can leverage this technology without needing to invest a huge amount of time or resources into developing it yourself.

We've also strived to put tools in the API to help you contextualize and understand the results in a way that makes sense for your application.

Now, the network we're talking about exposing here is in fact the same network we ourselves use to power the photo search experience. This is a network we've developed specifically to run efficiently on device without requiring any service side processing. We've also developed it to identify over a thousand different categories of objects.

Now, it's also important to note that this is a multi-label network capable of identifying multiple objects in a single image, in contrast to more typical mono-label networks that try to focus on identifying a single large central object in an image.

Now, as I talk about this new classification API, I think one of the first questions that comes to mind is what are the objects it can actually identify? Well, the set of objects that a classifier can predict is known as the taxonomy.

The taxonomy has a hierarchical structure with directional relationships between classes.

These relationships are based upon shared semantic meaning. For instance, a class like dog might have children like Beagle, Poodle, Husky, and other sub-breeds of dogs.

In this sense, a parent class tends to be more general while child classes are more specific instances of their parent.

You can of course see the entire taxonomy using ImageRequest.known Classifications.

Now, when we constructed the taxonomy, we had a few specific rules that we applied.

The first is that the classes must be visually identifiable.

That is, we avoid more abstract concepts like holiday or festival.

We also avoid any classes that might be considered controversial or offensive as well as those to do with proper names, nouns, excuse me, adjectives, or basic shapes.

Finally, we omit occupations, and this might seem odd at first. But consider the range of answers you'd get if we asked something like what does an engineer look like. There probably isn't a single concise description you could give that would apply to every engineer aside from sleep deprived and usually glued to a computer screen.

Let's take a look at the code you need to use in order to classify an image. So, as usual, you form an ImageRequestHandler to your source image. You then perform the VNClassifyImageRequest and retrieve your observations. Now, in this case, you actually get an array of observations, one for every class in the taxonomy and its associated confidence.

In a mono-label problem, you'd probably expect that these probabilities sum up to 1, but this is a multi-label classification network, and each prediction is an independent confidence associated with a particular class.

As such, they won't sum to 1, and they're meant to be compared within the same class, not across different classes. So we can't simply take the max amongst them in order to determine our final prediction.

You might be wondering then, how do I deal with so many classes and so many numbers. Well, there are a few key tools in the API that we've implemented to help you make sense of the result.

Now, in order to talk about these tools in the API, we first need to define some basic terms. The first is when you get a confidence for a class, we typically compare that to a class-specific threshold, which we refer to as an operating point. If the class confidence is above the threshold, then we say that class is present in the image. If the class confidence is below the class threshold, then we say that object is not present in the image.

In this sense, we want to pick thresholds such that objects with the target class typically have a confidence higher than the threshold, and images without the target class typically have a score lower than the threshold.

However, machine learning is not infallible, and there will be instances where the network is unsure and the confidence is proportionally lower. This can happen when objects are office gated, appear in odd lighting or at odd angles, for instance. So how do we pick our thresholds? Well, there are essentially three different regimes we can be in depending on our choice of threshold that yield three different kinds of searches.

To make this a little more concrete, let's say I have a library of images for which I've already performed classification and stored the results. Let's say in this particular case I'm looking for images of motorcycles.

Now, I want to pick my thresholds such that images with motorcycles typically have a confidence higher than this threshold and images without motorcycles typically have a score lower than this threshold. So, what happens if I just pick a low threshold. As you can see behind me, when I apply this low threshold, I do in fact get my motorcycle images, but I'm also getting these images of mopeds in the bottom right. And if my users are motorcycle enthusiasts, they might be a little annoyed with that result.

When we talk about a search that tries to maximize the percentage of the target class retrieved amongst the entire library, and isn't as concerned with these missed predictions where we say the motorcycle is present when it actually isn't. We are typically talking about a high recall search.

Now, I could maximize recall by simply returning as many images as possible, but I would get a huge number of these false predictions where I say my target class is present when it actually isn't, and so we need to find a more balanced point of recall to operate at.

Let's take a look at how I need to change my code in order to perform this high recall search.

So, here I have the same code snippet as before, but this time I'm performing a filtering with hasMinimumPrecision and a specific recall value. For each observation in my array of observations, the filter only retains it if the confidence associated with the class achieves the level of recall that I specified. Now, the actual operating point needed to determine this is going to be different for every class, and it's something we've determined based on our internal tests of how the network performs on every class in the taxonomy.

However, the filter handles this for you automatically. All you need to do is specify the level of recall you want to operate at.

So, we talked about a high recall search here, but what if I have an application that can't tolerate these false predictions where I'm saying motorcycles are present when they're not. That is, I want to be absolutely sure that the images I retrieve actually do contain a motorcycle. Well, let's come back to our library of images then and see what would happen if we applied the higher threshold. As you can see behind me, when I apply my high threshold, I do in fact only get motorcycle images, but I get far fewer images overall.

When we talk about a search that tries to maximize the percentage of the target class amongst the retrieved images and isn't as concerned with overlooking some of the more ambiguous images that actually do contain the target class, we are typically talking about a high precision search. Again, like with high recall, we need to find a more balanced operating point where I have an acceptable likelihood about my target class appearing in my results, but I'm not getting too few images.

So, let's take a look at how I need to modify my code in order to perform this high precision search. So here's the same code snippet, but this time my filtering is done with hasMinimumRecall and a precision value I've specified.

Again, I only retain the observation if the confidence associated with it achieves the level of precision that I specified. The actual threshold needed for this is going to be different for every class, but the filter handles that for me automatically. All I need to do is tell it the level of precision I want to operate at.

So we've talked about two different extremes here, one of high recall and one of high precision, but in practice, it can be better to find a balanced tradeoff between the two.

So, let's see how we can go about doing that, and in order to understand what's happening, I first need to introduce something known as the precision and recall curve. So, in practice, there is a tradeoff to be made where increasing one of precision and recall can lead to a decrease in the other. I can represent this tradeoff as a graph, where for each operating point I can compute the corresponding precision and recall. For instance, at the operating point at where I achieve a recall of 0.7, I find that I get a corresponding precision of 0.74. I can compute this for a multitude of operating points in order to form my full curve.

As I said before, I want to find a balance point along this curve that achieves the level of recall and precision that makes sense for my application.

So let's see how I need to change my code in order to accomplish it and how the precision and recall curve plays into that. So here I have a filtering with hasMinimumPrecision where I'm specifying the minimum precision and a recall value. When I specify a MinimumPrecision, I'm actually selecting an area along the graph that I want to operate within.

When I select a recall point with forRecall, I'm choosing a point along the curve that will be my operating point.

Now, if the operating point is in the valid region that I selected, then that is the threshold that the filter will apply when looking at that particular class.

If the operating point is not in the valid region, then there is no operating point that meets the constraints I stated, and the class will always be filtered out of my results.

In this sense, all you need to do is provide the level of precision and recall that you want to operate at, and the filter will determine the necessary thresholds for you automatically. So, to summarize, the observation I get back when performing image classification is actually an array of observations, one for every class in the taxonomy.

Because this is a multi-label problem, the confidences will not sum to 1. Instead, we have independent confidence values, one for every class between 0 to 1, and we need to understand precision and recall and how they apply to our specific use case in order to apply a filtering with hasMinimumPrecision or hasMinimumRecall that makes sense for our application. So, that concludes the portion on image classification. I'd like to switch gears and talk about a related topic, Image -- excuse me. Image Similarity.

When we talk about Image Similarity, what we really mean is a method to describe the content of an image and another method to compare those descriptions.

The most basic way in which I can describe the contents of an image is using the source pixels themselves.

That is, I can search for other images that have close to or exactly the same pixel values and retrieve them.

If I did a search in this fashion, however, it's extremely fragile, and it's easily fooled by small changes like rotations or lighting augmentations that drastically change the pixel values but not the semantic content in the image.

What I really want is a more high-level description of what the content of the image is, perhaps something like natural language. I could make use of the image classification API I was describing previously in order to extract a set of words that describe my image. I could then retrieve other images with a similar set of classifications. I might even combine this with something like word vectors to account for similar but not exactly matching words like cat and kitten. Well, if I performed a search like this, I might get similar objects in a very general sense, but the way in which those objects appear and the relationships between them could be very different.

As well, I would be limited by the taxonomy of my classifier. That is, any object that appeared in my image that wasn't in my classification networks taxonomy couldn't be expressed in a search like this.

What I really want is a high-level description of the objects that appear in the image that isn't fixated on the exact pixel values but still cares about them. I also want this to apply to any natural image and not just those within a specific taxonomy. As it turns out, this kind of representation learning is something that's naturally engendered in our classification network as part of its training process.

The upper layers of the network contain all of the salient information necessary to perform classification while discarding any redundant or unnecessary information that doesn't aid it in that task. We can make use of these upper layers then to act as our feature descriptor, and it's something we refer to as the feature print.

Now, the feature print is a vector that describes the content of the image that isn't constrained to a particular taxonomy, even the one that the classification network was trained on. It simply leverages what the network has learned about images during its training process.

If we look at these pairs of images, we can compare how similar their feature prints are, and the smaller the value is, the more similar the two images are in a semantic sense. We can see that even though the two images of the cats are visually dissimilar, they have a much more similar feature print than the visually similar pairs of different animals. To make this a little more concrete, let's go through a specific example. Let's say I have the source image on screen, and I want to find other semantically similar images to it.

I'm going to take a library of images and compute the feature print for each image and then retrieve those images with the most similar feature print to my source image. When I do it for this image of the gentleman in the coffee shop, I find I get other images of people in coffee shop and restaurant settings.

If I focus on a crop of the newspaper, however, I get other images of newspapers. And if I focus on the teapot, I get other images of teapots.

I'd like to now invite the Vision Team onstage to help me with a quick demonstration to expand a little more on how Image Similarity works.

Hello everyone. My name is Brett, and we have a really fun way to demonstrate Image Similarity for you today. We have very creatively called it the Image Similarity game.

And here is how you play. You draw something on a piece of paper, then ask a few friends to re-create your original as close as possible. So I will start by drawing the original.

Okay. Tap continue to scan it in as my original.

And then save.

Now, my team will act as contestants, and they will draw this as best as they can.

Now while they're drawing, I should tell you that this sample app is currently available to you now on the developer documentation website as sample code, and also, we are using the Vision kit document scanner to scan in our drawings, and you can learn more about that at our text recognition session.

Let's them give a few more seconds. Five, four, three, okay, I guess they're done. Okay. Let's bring them up and start scanning them in. Contestant number one.

Pretty good .

That might be a winner. Let's see contestant number two.

Still pretty good. Nicely done. Contestant number three please.

I think that's pretty good. And contestant number four.

Well, I don't know about that, but we'll see how it goes. All right. So let's save those, and we find out that the winner is contestant number one. Congratulations. Now I can swipe over, and we can see that the faces are more semantically similar that way. They are closer to the original while the tree is semantically different, it was much further away. And that is the Image Similarity game, and background check to Rohan. Thanks everyone. I want to take a quick look at a snippet from that demo application to show how we determined the winning contestant.

So here I have the portion of the code that compares each of the contestant's drawings feature print to Brett's drawing's feature print. Now, I extracted the contestant's feature print with a function we have defined in the application called featureprintObservationForImage. Once I have each feature print, I then need to determine how similar it was to the original drawing, and I can do that using computeDistance, which returns me a floating-point value. Now, the smaller the floating-point value, the more similar the two images are. And so, once I've determined this for every contestant, I simply need to sort them in order to determine the winner.

Well, this concludes the portion on Image Similarity. I'd now like to hand the mic over to Sergey to talk about some of the changes coming to Face Technologies. Good morning everybody. My name is Sergey Kamensky. I'm a software engineer on the Vision Framework Team. I'm excited to share with you today even more new features coming to the Framework this year. Let's talk about Face Technology first. Remember, two years ago when we introduced Vision Framework, we also talked about Face Landmark detector. This year, we're coming with a new revision for this algorithm. So, what are the changes? Well, first, we now have 76-point cancellation, and this is versus 65-point cancellation as we had before. The 76-point cancellation gives us a greater density to represent different face regions.

Second, we now report confidence score per landmark point, and this is versus a single average confidence score, as we reported before. But the biggest improvement comes in the pupil detection. As you can see, the image on the right-hand side has pupils detected with much better accuracy. Let's take a look at the client code sample.

This code snippet will repeat throughout the presentation so the first time we're going to go line by line. Also, I use for in my samples. If this is just to simplify the slides, when you develop your apps, you probably should use proper error handling to avoid unwanted boundary conditions. Let's get back to the sample.

In order to get your facial landmarks, first you need to create a DetectFaceLandmarksRequest. Then, you need to create ImageRequestHandler, passing the image into it the image that needs to be processed, and then you need to use that request handler to process your request. Finally, you need to look at the results. The results for everything that this human face related in Vision Framework will come in forms of face observations. Face observation derives some detected object observation. It inherits bounding box property, and it also adds several other properties on its level to describe human face.

This time we'll be interested in the landmarks property. The landmarks property is of FaceLandmarks2D class. FaceLandmarks2D class consists of the confidence score. This is the average single average confidence score for the entire set and multiple face regions where each face region is represented by FaceLandmarksRegion2D class. Let's take a closer look at the properties of this class. First is pointCount. PointCount will tell you how many points represent a particular face region. This property will a different value depending how you configure your request, with 65-point cancellation or 76-point cancellation.

The normalizedPoints property will represent the actual landmarks point, and the precisionEstimatesPerPoint will represent the actual confidence score for teach landmark point.

Let's take a look at the codes needed. This is the same code snippet as in the previous slide, but now we're going to look at it from a slightly different perspective. We want to see how revisioning of the algorithm works in Vision Framework. If you take this code snippet and recompile it with the last , what you will get is that the request object will be configured as follows: the revision property will be set to revision number 2, and the cancellation property will be set to cancellation of 65 points. Technically, we didn't have cancellation property last year, but if we did, we could have set it to a single value only. Now, if on the other hand you take the same code snippet and recompile it with the current , what you will get is that the revision property will be set to revision number 3, and the cancellation property will be set to cancellation 76 points.

This actually represents the philosophy of how Vision Framework handles revisions of algorithms by default. If you don't specify a revision, what we will do is, we will give the latest supported by the SDK your code is compiled and linked against. Of course, we'll always recommend to set those properties explicitly. This is just to guarantee deterministic behavior in the future. Let's take a new metric that we developed this year, Face Capture Quality. There are two images on the screen. You can clearly see that one image was captured with better lighting and focusing conditions.

We wanted to develop the metric that looks at the image as a whole and gives you one score back saying how bad or good the capture quality was. As a result, we came up with a Face Capture Quality metric.

We trained our models for this metric in such a way so they tend to score lower if the image was captured with low light or bad focus, or for example, if a person had negative expressions.

If we run this metric on these two images, we will get our scores back. These are floating-point numbers. You can compare them against each other, and you can say that the image that scored higher is the image that was captured with better quality. Let's take a look at the code sample.

This is very similar to what we saw just a couple of slides ago, with the differences being in the request type and the results. Since we still with C1 faces, we're going to get our face observation back, but now we're going to look at a different property of the face observation, Face Capture Quality property. Let's take a look at the broader example. Let's say I have a sequence of images that could have been obtained by using the burst mode on the selfie camera or in the photo burst, for example. And you will ask yourself a question. Which image was captured with the best quality? What you can do now, you can run our algorithm on each image, assign scores, rank them, and the image that apps on the most light is the image that was captured with the best quality. Let's try to understand how we can interpret the results that are coming from the Face Capture Quality metric. I have two sequences of images on the slide. Each sequence is of the same person, and each sequence is represented by the images that scores lowest and the highest in the sequence with respect to Face Capture Quality.

What can we say about these ranges? Well, there is some overlapping region, but there are some also regions that belong to one and don't belong to the other. If you had yet another sequence, it could have happened that there was no overlapping region at all.

The point I'm trying to make here is that the Face Capture Quality should not be compared against a threshold.

In this particular example, if I picked 0.52, I would have missed all the images on the left, and I would pretty much can get any image that's just past the midpoint on the right.

But then what is Face Capture Quality? We define Face Capture Quality is a comparative or ranking measure of the same subject. Now, comparative and same are the key words in this sentence. If you're thinking, cool, I have this great new metric, I'm going to develop my beauty contest app.

Probably not a good idea. In a beauty contest app, you would have to compare faces of different people, and that's not what this metric was developed and designed for.

And that's Face Technology.

Let's take a look at the new detectors we're adding this year.

We're introducing Human Detector that detects human upper body that consists of human head and torso and also a pet detector, an Animal Detector that detects cats and dogs. The Animal Detector gives you bounding box back, and in addition to bounding boxes it gives you also a label saying which animal was detected.

Let's take a look at the client code sample.

Two snippets, one for Human Detector, one for Animal Detector. Very similar to what we had before. Again, the differences are in the request types that you create and in the results. Now, for Human Detector, all we care about is the bounding box. So, we use for that DetectedObjectObservation.

For the Animal Detector on the other hand, we also need the label, so we use RecognizedObjectObservation that derives from detected object observation. It inherits bounding box, but it also adds a label property on the . And that's new detectors. Let's take a look at what's new in tracking this year.

We're coming up with a new revision for the Tracker. The changes are, we have improvements in the bounding boxes expansion area.

We can now handle better occlusions. We are machine learning based this time. And we can run with low power consumption on multiple devices. Let's take a look at a sample. I have a mini video clip where a man is running in the forest, and he appears sometimes behind the trees. As you can see, the tracker is able to successfully recapture the tracked object and keep going with the tracking sequence.

Thank you.

Let's take a look at the client code sample. This is exactly the same snippet that we showed last year. It represents probably the simplest tracking sequence you can imagine. It tracks your object of interest for five consecutive frames.

I want to go line-by-line, but I want to emphasize two points here. First is we use our SequenceRequestHandler.

That is as opposite to ImageRequestHandler as we have used so far throughout the presentation. SequenceRequestHandler is used in Vision when you work with a sequence of frames and you need to cache some information from frame to frame to frame.

Second point is when you implement your tracking sequence, you need to get your results from iteration number n and feed it as an input to a duration number n plus 1. Of course, if you recompiled this quote with the current SDK, the revision of the request will be set to revision number 2 by default. But we also recommend to set it explicitly. And that's the tracking.

Let's take a look at the news with respect to Vision and Core ML integration. Last year, we presented integration with Vision and Core ML, and we showed how you can run Core ML models through Vision API. The advantage of doing that was that you can use 1 over 5 different overloads of the image request handler to translate the image that you have in your hand to the image type, size, and color scheme that the Core ML model requires.

We will run the inference for you, and we'll pack the outputs or results coming from Core ML model into Vision observations.

Now, if you have a different task in mind, for example, if you want to do image style transfer, you need to have at least two images, the image content and the image style. You may also need to have some mixed ratio saying how much of a style needs to be applied on the content. So, I have three parameters now.

Well, this year we're going to introduce API where we can use multiple inputs through Vision to Core ML, and that's including multi-image inputs. Also, on the output section, this sample shows only one output. But, for example, if you had more than one, especially if you have more than one of the same type, it's hard to distinguish them when they come in forms of observation later on. So, what we do this year, we introduce a new field in the observation that maps exactly to the name that shows up here in the output section.

Let's take a look at the inputs and outputs. We will use them in the next slide.

This is the code snippet that represents how to use Core ML through Vision.

The highlighted sections show what's new this year. Let's keep them for now, and we'll go over the code, and we'll return to them later.

In order to run Core ML through Vision, first you need to log your Core ML model.

Then, you need to create Vision CoreMLmodel wrapper around it. Then, you need to create Vision CoreMLRequest and pass in that wrapper.

Then you create ImageRequestHandler, you process your request, and you look at the results.

Now, with the new API that we added this year, that only image that you could use last year is the default or the main image is the image that is passing to ImageRequestHandler, but that's also the image whose name needs to be assigned to input feature name field of the CoreMLModel wrapper.

All other parameters whether images or not will have to be passed through feature provider property of the CoreMLModel wrapper. As you can see, image style and mixed ratio are passed in that way.

Finally, when you look at the results, you can look at the feature name property of the observation that comes out, and you can compare it in this case against image result. That's exactly the name that appears in the output section of Core ML, and that way you can process your results accordingly.

This slide actually concludes our presentation for today. For more information you can refer to the links on the slide. Thank you, and have a great rest of your WWDC. [ Applause ]