Improve your prompts by hill-climbing with Evaluations

HI! My name is Marcus, a manager on the Evaluations framework team.

I'm excited to show you how to use Evaluations to improve your intelligence-powered feature. As you probably know by now, using AI in your app is a powerful way to provide new levels of personalization to your users. This technology can add a level of depth to your app that was not previously possible with traditional software. However, it's also a challenge to know whether or not your intelligence-powered feature behaves as you'd expect in all cases. To help with this, we're releasing the Evaluations framework to provide you with the tools you need to ship with confidence. Shipping with confidence takes more than just a framework. The Evaluations framework also allows you to hill climb, which is a process of iteratively improving the quality of your feature using the scores of your evaluation as a guide. Hill-climbing starts with development, that is making some change you want to measure against your existing feature.

Once all your changes are made, you then need to run the evaluation. And see if the results have passed your expectations.

From there, you analyze the results to understand how your feature could be further improved.

Leveraging the hill-climbing process is a great way to systematically improve your feature, but effective hill-climbing takes a little bit more than just following the loop. It also takes a little bit of… Science! So, in this video, I'll walk you through how to improve prompts by following the hill-climbing loop, while incorporating some scientific thinking along the way.

Next, I'll walk you through how to conduct comparative Evaluations to make the process of hill-climbing easier. And finally, we'll go beyond changing prompts, by improving other aspects of an intelligence-powered feature.

But, before we go any further, this video is about the process of hill-climbing an existing evaluation. That means you've already written the foundations of an evaluation pipeline, which provides a wholistic understanding of the strengths and weaknesses of your intelligence-powered feature.

If you're unfamiliar with how to do that, go check out our other video, "Meet the Evaluations framework". It covers everything you need to know in order to build a great evaluation pipeline.

With that covered, let's get started.

In the "Meet the Evaluations framework" video we introduced you to Book Tracker. In case you forgot, Book Tracker allows readers to catalog and review books.

Recently, I've been reading a lot of classics, so I've added the books to my catalog. In fact, I just finished reading "Treasure Island"! It's such a thought-provoking read that covers the tension between loyalty and betrayal.

One of Book Tracker's new features is a tagging service, which uses a model to generate tags based on the readers review. While the tags for this review cover the general themes of the book, I feel like something is missing. I would have expected to see tags like "tense" or "morally grey", which speak to the themes of the story. The tags generated for "Little Women" had a similar problem. Tags like poignant has more to do with how the reader felt than the contents of the book. Emotion in book reviews is great, but it should not make it into the list of tags. Also, a tag like quiet-steadiness, which is pulled directly from the review, isn't going to be a very useful tag when I want to search my library later.

It seems like Book Tracker's tag generator isn't as good as I think it could be.

Fortunately, when my colleagues wrote this feature, they wrote an Evaluation, which measures the tags against a set of criteria.

And here is my Evaluation for Book Tracker's book tags. I'm particularly interested in how we are judging the quality of my tags so I'll scroll down and take a look.

The qualitative aspects of my app are captured by the score dimensions type.

Relevance tracks the how well the tags represent information about the book's plot, theme, or other relevant information. Usefulness, measures how good the tags are as search terms.

The ModelJudgeEvaluator uses the score dimensions and a prompt to generate a score for each set of tags.

My plan is to add these two books to my Evaluation, and review the tags I get back.

It will also be a good opportunity to see how my ratings stack up against the ratings of my model judge.

Wanting to improve some part of an intelligence powered feature is how the hill-climbing process starts. So, to start the loop you begin in the development phase. Here, you make any changes you need to feature as well as your Evaluation.

In this case, I'll add my review of "Treasure Island" to my Evaluation dataset. And I've done the same thing for "Little Women".

With those two entries now in my dataset, I now want to run my Evaluation and see what tags the model produces and how the judge scores them.

But, to get there we need to first ask if the Evaluation we ran met our expectations.

As a reminder, your expectations can be defined using Swift Testing's expect macro.

That way, you can tell if your expectations are met by whether or not your tests pass.

In this case, my Evaluation met all of my expectations; however, because I know the tags aren't as good as I'd like them to be, I need to investigate further.

And that brings us to the analyze phase.

Xcode's new evaluations report gives me in depth information about my last Evaluation run.

To see more details I can click on my BookTaggingEvaluation run.

This brings up the Evaluation detail view.

On the top are our aggregate metric charts.

And below is our table of results.

What I'd like to do now is compare the response of the model to the list of expected tags I generated before. I can do that by opening up the Assistant Editor.

And now I can see detailed information about the tags generated for each item in my dataset.

I want to focus on the difference between what the model generated and what I expected.

I can review that in detail in this table. The collection of terms isn't bad but it leaves out a number of key details from the story which the user might want to search for.

Because of that, I would have scored these tags a 4 for relevance and a 2 for usefulness.

My model judge also gave the tags a relevance score of 4 which is great, but it also gave usefulness a score of 4, which isn't right.

I should see if I feel the same way about the tags for my review of "Little Women".

The tags don't really contain all the useful information that I'd expect.

And it turns out I disagree with the judge's scores here as well.

Once again, I think relevance should be a 4 and usefulness should be a 2.

Doing this analysis has made it clear to me that there is a discrepancy between how my model judge and I rate tags.

This discrepancy between model and human is known as drift, and it is a problem faced by all developers trying to evaluate intelligent features. Here's why. Say I have an evaluation with 10 samples. I then ask a model judge and a person to rate each sample.

The model and person then give their ratings on a scale from 1 to 4, and at the end we average those scores to build an aggregate.

If the model and the human tend to disagree in their ratings, then their average scores will diverge from one another, hence the name drift.

As your data set continues to grow and grow the drift will get wider and wider. At which point, it'll be hard for you to know whether or not your feature is being properly evaluated.

To help with this, you can align your judge to a person's expert opinion. Now that we know drift is a problem, we need a way to know how much our model judge has drifted from our expert ratings.

One way to accomplish this would be to line up the ratings of the expert and mark where the two match. You can then use this to generate a percentage. This percentage is called accuracy, and it is a great way to measure alignment if every value in your scoring scale is equally likely to appear. However, it's more likely that your dataset will contain values that have an uneven distribution of scores. Think about it, datasets often contain examples of high quality output.

Therefore it is often the case that a human rater is likely to rate items in the dataset with higher scores.

If a model then happens to judge your smaller dataset with high scores, it may seem like the two are aligned. But then when unleashed on a larger dataset with more variations in scores, it's tendency to score high will still result in drift.

So we need an alternative to accuracy, one that accounts for the weighted nature of our dataset and the chance that the model might guess the right answer. Fortunately there is a solution! Cohen's kappa coefficient is a mathematical formula made popular by statistician and psychologist Jacob Cohen in 1960.

Cohen's kappa measures alignment, that is how often do two raters agree.

To do that, we need to know what percentage of the time the raters agreed, better known as accuracy.

And this is exactly what the accuracy metric from before was calculating. But now we need to calculate a new value. Coincidence, which represents the chance that one rater might get lucky and happen to align.

This luck is then weighted based on the chances certain answers are more likely to appear. So now the question is, how do you calculate it? To calculate alignment, we start with our accuracy score.

From the accuracy score we subtract the possibility of two raters randomly agreeing.

Finally, we divide the difference by the inverse of random agreement, namely the chance that the two raters intentionally agreed.

The result of that gives us alignment.

Cohen's kappa is a powerful way to measure the alignment between a model judge and your expert opinion. I can use this to hill climb the alignment scores between me and my model judge.

So now, we start back at the beginning of hill-climbing loop in the develop phase.

To do this, I am going to set up an evaluation to compare my ratings against my judge and produce and alignment score. To do that, I need to write an evaluation, which is made up of four components. First is my dataset.

Then the subject of my evaluation. Then, I need to define my evaluators. And finally, I need to aggregate my results. So let's start with the dataset.

For this evaluation to work properly, both my model judge and I need to evaluate the exact same dataset. In this case the model judge reviews tags, so I need to produce a common set of tags for the judge and I to review. And I have just the perfect dataset.

My evaluation from before contains a collection of reviews and tags.

Because I ran this evaluation in a test, Xcode generated an attachment containing all the of evaluation data that was generated. I can retrieve that attachment and extract summary and tag pairs. Now, with the summary and tag pairs extracted, I need to add my ratings. After that, I can pass the contents of this file as the input to my evaluation.

Next, I need to capture the subject of my evaluation. Normally, the subject method is for calling API related to your feature, but since the generated model responses are part of our dataset, we can simply return the already generated tags. Now, I need to define my evaluators. As you might have guessed, my evaluator is the exact same model judge evaluator as in our book tags evaluation. This is where the judge provides its rating.

Finally, I need to aggregate my results.

Here is where we compare my ratings against the judge's. To do that, we need to calculate Cohen's kappa, which I can do that with a custom aggregation method. In addition to just Cohen's kappa, I'll also calculate the mean and standard deviation of each score dimension. This will be helpful to know if the scores of the judge are going up or down. Now, I can setup my test with my evaluation. For this test, I've set an expectation that my ratings and the judges ratings should produce an alignment score of 0.6. We've chosen this number because according to statisticians, an alignment score of 0.6 represents a meaningful level of agreement. Now, it's time to evaluate and get a baseline for our alignment. And then determine if my evaluation has passed my expectations.

It appears that the tests failed, which means my expectations weren't met. So once again, it's time to analyze the results in detail. I now know that my alignment scores didn't match my expectations. I can now go to the evaluation report to get more information. As I expected, the scores for both usefulness and relevance are quite low, meaning my model judge and I aren't aligned.

Now, I want to get more information about how each sample in my dataset performed. To do that, I need to open the assistant and view the results in detail.

As I scanned through the results, this review of "Frankenstein" caught my eye. I can see a pretty large discrepancy between my rating of the tags and the judge's.

It seems like our judge thinks tags like self-help and self-improvement are relevant to the story. Also psychological is an okay search term, but probably not a term a user is likely to search for.

I then started looking through other items in my dataset that had a similar problem and came across this review of "The Ramakien".

The judge and I agree that these collection of terms are helpful and relevant to the contents of the book.

Where we disagree is on usefulness.

Terms like visual-dimension and quaint-dignity are way too specific.

So what's the problem here? I believe the model doesn't have enough knowledge on it's own to distinguish between a good tag and bad one. That's likely because the prompt of my judge doesn't provide enough context. To do that, I need to develop a new prompt. That way I can compare the alignment scores of my current prompt against my scores of my new prompt. Fortunately, in Xcode 27, we've made so you can compare the results of two evaluations against each other. When doing comparisons, some scientific thinking can go a long way. In a science experiment, you have two groups. The control group, which represents the baseline and the experimental group which represents the change we are trying to compare against. We can think of the two versions of our instructions in the same way, where the control group is represented by our base prompt and our experimental group is represented by our newly changed prompt. I now need to create a second version of our evaluation with an experimental prompt. For our baseline, we will use the same evaluation with the same model judge prompt as before. For our experimental prompt, I've written a more thorough description about how to judge the set of tags.

It starts by providing the judge context about the app and what it's about to be judging. Then it gives examples of good tags.

As well as ways to identify bad tags.

With both prompts written, I can add both evaluations to a test suite, which will run both evaluations. So, I'll run that suite now and compare the results. With the evaluation now finished I can return to the evaluation report.

Looks like my alignment scores for relevance improved. While my alignment score for usefulness dropped considerably.

Balancing tradeoffs like this are tricky so I need to think carefully how to proceed.

But before in depth analysis comes checking if we passed.

And my test confirms the obvious, we haven't.

After thinking about it further, I am going to keep this prompt change and focus the next round of iteration on improving my usefulness score. Therefore, the most effective way to review my results is to compare the usefulness scores of both judges against one another. To do that, I can use the new comparison view in the evaluation report.

From the evaluation report I can open the comparison button and open my baseline evaluation.

Here, I can review the scores of the two prompts side by side.

One thing that jumped out to me immediately is the discrepancy between usefulness scores of this review of "Picture of Dorian Gray". It seems to me that the model may be judging too harshly on usefulness. The usefulness column of the experimental evaluation seems to corroborate my guess.

I noticed that all the scores are either a 3 or 2, which is way too harsh.

I think what could help here is being more specific about how to grade each scoring dimension.

To do that, I'll need to make some changes to my experimental evaluation.

But before I can make changes to the experimental evaluation, I applied the new prompt from my experimental evaluation into my baseline. This ensures there's only one different variable.

Namely, the changes to my scoring dimensions. For relevance, I've provided a slightly longer description which emphasizes the need for a genre tag.

And here is the one for usefulness. Which emphasizes being more critical of overly specific tags.

And once again I'll wait for my evaluation to run.

And the scores both improved greatly over the baseline. It looks like these specific scoring dimensions are going to be a lot more helpful.

But, we've still not quite hit our alignment goals. So now, I need to do another comparison to see where we might be able to make improvements. To analyze further, I've gone back to the experimental evaluation. I want to review the results in detail so I'll bring up the assistant view. Thumbing through the results brought me to the review of "Moby Dick".

My relevance score is starting to align.

But my usefulness score could still use some work.

While some results are looking promising, others are still way off. This review of "Frankenstein" continues to give our judge trouble.

What I think our judge needs now is some examples of the way I judge things, which should give it a pattern for how to judge according to my scale.

That means we need another round of hill-climbing.

I've already added the new score dimensions to my baseline evaluation. Now, I've reworked my main judge prompt to give it more detail about the goal of the tag generation feature to help ground the model in the problem space.

From there, I've written out a number of examples for the model to use as a guideline for reviewing. I've made sure to only give the model a few examples. By giving it a longer list I am prone to overfit the alignment score, which would make it hard to tell if my judge is actually aligned with me. Now that it's a fair comparison, I need to run my evaluation and view the results. And now finally my scores are over my expected value! Which means I've finally passed and can exit out of the loop! This now means I can be confident that when my model judge provides ratings, I can confidently say that the tags are good or bad according my standards. That means I can now put my judge to work on evaluating Book Tracker's Book Tagging Service. So far we've seen how to hill climb on prompts, making them incrementally better and better, now I'd like to show you how to improve your feature through something other than your prompts.

To generate its tags, Book Tracker uses the on-device model. We use it because readers tend to be in all kinds of places when cataloging books, so using the on-device model ensures they can generate tags no matter where they are.

What I want to do is give the model some more context about the book it's generating tags for. I think the additional context will help the model generate more relevant and useful tags.

Better still, Book Tracker already has the data needed for this because we store the author's name and book title when they write their review. So, to help the tag generator, I've created a tool to get additional information on the book, which provides the book title and author if they are available. Adding this tool is a form of hill-climbing because we are attempting to improve the quality of our feature through an incremental change.

For this evaluation we will use the book tagging Evaluation, now with an improved model judge.

But I need a way to compare the quality of my feature without the tool to the quality of my feature with it. So to do that, I'll need to make a change to Book Tagging Service BookTaggingService now takes a list of tools as input. I also set the default to an empty array so my existing evaluation won't need any changes. But now I need to write a new evaluation to compare the service with the tool to the service without the tool.

Here is the new evaluation I wrote. It's exactly the same as the other evaluation. The only difference is I now pass my new lookup tool in the tools array.

So all I have to do is define two instances of my evaluation. One without the tool and one with it.

And now, let's evaluate it and determine if I'm ready to ship.

Well, my service which uses tools met all my expectations, so things are looking good. But, my dataset for Book Tracker contains only 13 book and review pairs, that doesn't cover the wide variety of books and reviews a user might submit for tagging.

In addition, I was looking at the results of the evaluation of my service with the tool. I can see that the service with the tool is performing better, however it does seem like my tool isn't being called in all the places I think it needs to. What I really need is a way to tell whether or not my tool has been called in the right situations. Fortunately, the Evaluations framework can help with both of those problems. To learn more about our APIs for evaluating tool usage and generating comprehensive datasets, take a look at the "Create robust evaluations for agentic apps" video. There, you'll learn about tool call Evaluators and how to use the Sample Generator API to test the wide variety of uses cases your app might see. Before we wrap up, I'd like to recap what we covered today.

Hill-climbing works best when you focus on making one change at a time. To do this, treat every iteration of the loop like a science experiment.

Being able to isolate your changes will help you to understand how each part of your feature contributes to the overall quality.

Knowing how each part works individually will also help you to know where you might need to make changes to resolve a bug or unwanted pattern later down the line.

Second, this process takes time. Not every change you make will result in positive change. However, failed experiments tell you just as much as successful ones.

Third, good experiments require creativity. In an intelligent feature there are so many things you can change.

In your feature you can change the instructions, the tools, as well as the model or models you use to generate responses.

On the evaluation side you can change the dataset, aggregation methods, and even the evaluators themselves. Everything is fair game. Make sure to consider all of these when thinking about how to hill climb.

Finally, watch out for drift. It can feel a bit meta to evaluate your evaluators but a well tuned model evaluator will save you time in the long run. Models can generate ratings much faster than humans can. So by keeping them aligned, you get useful signal as your dataset grows to cover more and more use cases.

If you want to learn more about what we've covered here today, you can review the Book Tracker app I've been using as well as the evaluations for aligning the model judge. You can also get a comprehensive rundown of all our new APIs on the developer documentation website. Thank you for taking the time to learn about how to improve your evaluation scores by hill-climbing. Your dedication will pay off as you deliver high quality experiences to your users. Thanks for watching and happy hill-climbing!