Analyze HTTP traffic in Instruments

Welcome! My name is Kacper. I'm a Performance Tools Engineer at Apple, and today, together with Sergio, we'll be talking about the new HTTP Traffic Instrument available in Instruments 13. This Instrument, contained in the Network template, allows you to inspect HTTP traffic coming from your application through the Apple Networking stack.

This approach comes with multiple benefits. It just works on all Apple devices. The entire traffic going through the URL loading system is exposed, even the ones sent with the new HTTP/3 protocol or over VPN. Because of the system integration, it attributes traffic to processes running on it, and since it's instrumenting Apple Networking frameworks, it even reveals requests hitting the on-disk cache or networking errors. All of this exposed in the context of higher-level API concepts that you are familiar with, such as URLSessions and URLSessionTasks. This tool will help you understand how your usage of the API translates to the lifetime of network requests. In this hands-on session, we will first get you familiar with how the Instruments UI reflects the API concepts you are using.

After this quick introduction, we will transition to four demos that will illustrate how you can use the tool to detect both performance and correctness problems. And even if your app seems to work flawlessly, you will learn how to verify that it does what you think by auditing the traffic coming from it. Let's get started with how the Networking API maps to the Instruments visualization. This is how the HTTP Traffic trace is displayed in Instruments when I recorded my system traffic using the Network template. Navigation is structured around the track hierarchy, and that's what we will cover in detail first. The HTTP Traffic Instrument located at the top level shows you an overview of how many URLSession tasks were running in your trace at any given time, ideal for detecting spots with increased HTTP Traffic activity in your app's lifetime. The next hierarchy level shows a breakdown of activity by process. In addition to traffic from all of the debuggable processes, it allows you to inspect background traffic initiated by them. Contained below each process are all URLSessions used by it. And these correspond to the URLSession objects you create in code. The graph at this level allows you to inspect all individual task intervals. To get a better mapping between your session objects and the visualization, you can name them in code by setting the sessionDescription property on a session instance.

On the last level, the traffic is broken down by requested domains. Graphs on this level show more details about tasks, including individual transactions that make up the tasks and their states.

To get a better understanding of what tasks and transactions are, let's analyze an example.

Here are a few tasks that are loading data from the selected domain. Let's focus on one of them to analyze the structure of a task.

This single task interval has a lot of information. We can represent this in a more abstract manner to understand how Instruments visualization maps to the API being used.

At the top level, we have a task object. A task is made up of one or more transactions.

A transaction is a pair of an HTTP request and the corresponding response.

The task level is the representation of how your code interacts with the API of the URL Loading System. When you create a task and call resume on it, the task interval starts. And it ends right before your completion block is called.

Each task can be given a semantic name using the taskDescription property, which will be used to label the interval in Instruments. We also show the task identifier as part of the task label. You can use it to cross-reference the task with other data. If your task finishes with an error, its description will be presented on the interval label for easier debugging. As we mentioned before, a task can be made up of multiple transactions. Let's talk about these now. Here we have a task to load the start page of apple.com. However, this URL is not the canonical URL. The task requests apple.com, but the preferred domain is www.apple.com. When we create this task, the URL loading system initially creates a request to apple.com. Shortly after, it receives a redirect response from the server, stating that the preferred URL is actually www.apple.com.

By default, we follow redirects, so instead of returning the 301 response, the URL loading system will create a new transaction to now load the preferred URL. The response from this second, successful transaction is what is returned back to the task.

As mentioned before, a transaction represents the combination of the HTTP request and response. It aligns with what URLSession does under the hood to handle your task and contains all of the information of the HTTP layer, like the requested URL, information about the transferred data, and much more.

Just like for the task, the transaction label gives you an overview of the transaction. Mainly, you get information about the request and the response.

The track hierarchy tells you the domain that is requested, while you can find the path and query on the label itself.

In addition to that, the interval label displays the HTTP version, the HTTP Method, and whether the request sent an Authorization or a Cookie header. These are often useful to understand authentication flows at a glance.

For the response, you get the status code, whether the response contained a cookie, and the content type of the response. How long the request and the response took, as well as more detailed timing information about other work that is part of the transaction is captured by the transaction states. Let's analyze them in the context of a containing task. The start of the transaction is the point in time when the URL Loading System creates the transaction for making this request. It first checks whether we have a valid cached response already. And if that's not the case, it will try to schedule the request on a connection.

Next, the transaction may have to wait a bit in the Blocked state, waiting for an available connection.

The Sending Request state starts when the transaction is finally handled by a connection. It ends once we send the last byte of the request onto the network. Next, the transaction goes into an idle Waiting for Response state, followed by Receiving Response, which will track the span from the first to the last byte received from the server.

The whole transaction will complete shortly after the last byte is received, once the URL Loading system has determined whether this was a successful response. In practice, the cache lookup and sending state for a GET request are usually much shorter, so it's more likely to appear like this.

To show you some practical examples, I would like to hand over to my colleague, Sergio. He will walk you through an app he recently started developing to illustrate how the HTTP Instrument can help you with fixing performance and correctness issues. Thanks, Kacper. Hello, everyone. My name is Sergio Lopez, and I've been working on this app for dog lovers. Think of it like a social media platform but for dog pictures... only! People can post images of dogs and you get a stream of the most recent uploads! So when I open the app, it loads several new dog images, but I noticed that it takes quite some time for them to finish loading.

Let's profile the app with the new HTTP Traffic instrument to help us improve this situation. In the "Product" menu, I'll choose the "Profile" option to profile my app in Instruments.

This will build my app in the release configuration, to ensure I'm profiling my app as it would run for my users, with all optimizations turned on. Once the build is finished, Instruments will launch automatically. Upon starting, the standard template chooser of Instruments is displayed.

In our case, I wanna choose the Network template on the bottom left, which gives us more information about general network connections my app makes, but also contains the new HTTP Tracing functionality.

The track area now contains two tracks, one for each instrument. The bottom track is the existing Network Connections instrument, and the top track is the new HTTP Traffic instrument. We'll focus on this new instrument today. All I need to do now is hit "record." Instruments will then start my app and start recording.

Before you can use this tool, you need to confirm that you understand the implications of capturing the networking traffic. It's very powerful, especially if you record all processes. The data captured includes everything that is sent, which may be personal and sensitive information, even up to user credentials. So, you should be very careful with the resulting trace files, and we want you to be aware. So let me confirm this.

The app was launched and the images were slow to load.

I will now stop the recording.

Let's zoom in to the data we recorded by using Option-click and dragging over the area covering our HTTP traffic.

Clicking on the disclosure indicator in the "HTTP Traffic" track on the top left will reveal the full track hierarchy that Kacper described earlier.

I'll also increase the track height to show all intervals.

At the top, there's the first task that queries the server for the list of images, which appear on the "Latest" section of the app.

When this task completes, we create a new task to load a thumbnail for every image on the list that we received.

I will now click-drag over the area covering the time frame it took to fetch the list of images, followed by the many requests to retrieve each individual image.

By click-dragging over this area, a tool tip will be displayed, showing the duration of the selected time range. Overall, it took more than 7 seconds to finish loading the initial screen.

The first few images load fairly quick. But as I scroll down, tasks that were started later took longer to complete, as noted by the increasing blocked states in purple. Seems like a congestion issue, where we have too many requests in parallel. Let's investigate one of the later tasks.

By hovering over the task, the tool tip shows us the duration of the task and any of the child intervals we are hovering over. This task was blocked for the majority of the time.

To understand why it was blocked, let's switch the track display to the "HTTP Transactions by Connection" view.

In the track sidebar on the left, under the domain name, there's a downwards arrow we can click to switch the track display.

Currently, we are drawing "Tasks." Let's switch to displaying "HTTP Transactions by Connection." This view will only display the transactions, and instead of grouping them by task, we can now find out which connection they got scheduled on.

The transactions are grouped by the connection they used. Overall, there were six connections available to handle these transactions. Let's analyze the transactions issued on Connection 1 and investigate some of the thumbnail loading transactions further. From the top down, it's noticeable that each transaction is taking longer to complete. The purple blocked state for each successive transaction is increasing. In fact, there's a pretty clear staircase pattern here.

Each transaction is blocked, until the previous transaction on the same connection has finished. Only then can it send its request. This pattern repeats for each subsequent transaction.

This is called "Head of Line Blocking" and is one of the problems of using HTTP/1.

The frustrating part is that these transactions aren't doing anything for the majority of the time. Instead, they spend most of their time blocked or waiting for the response from the server. We could be sending another request for the next transaction in line while waiting for the response of a previous transaction on the same connection, but that's not supported by HTTP/1. Head-of-line blocking is one of the main limitations of HTTP/1, and one of the main improvements of HTTP/2 is to avoid that effect by multiplexing several requests to the same server onto a single connection.

In HTTP/2, we can actually start sending a second request while the first one is waiting for its response. Your app does not need to do anything to support it. All Apple platforms support HTTP/2, and starting in iOS 15 and macOS Monterey, HTTP/3 is supported as well. The client will pick the most modern HTTP version the server supports. If you wanna learn more about the differences between HTTP/1 and HTTP/2, and the additional benefits HTTP/3 provides, please watch the "Accelerate networking with HTTP/3 and QUIC" session. I've taken this trace, showed it to our server folks, and managed to convince them that we should really support HTTP/2. Now, let's run my app with the new server enhancements.

Wow, this already feels faster! Let's confirm this with Instruments. So here's a trace I recorded after we turned on server support for HTTP/2. In the domain-specific track, none of our thumbnail-loading tasks seem to be blocked anymore for an extended amount of time. That's good! Let's switch to the "HTTP Transactions by Connection" view again.

The first thing we notice is that there is only one connection. This is because we no longer need multiple connections to send concurrent requests, which also means we only need to pay the connection setup cost once. Focusing on the individual thumbnail-loading transactions, we notice that they basically spend no time in the "blocked" state. In fact, the amount of time is so small that it's not visible at this zoom level. Eventually all transactions finish sending their requests and are left waiting for a response. As I scroll down, we can notice that responses are making progress at the same time.

All in all, we are done with all requests in under 3 seconds. This is twice as fast as before. Now that I've talked to the server folks and switched from HTTP/1.1 to HTTP/2, our images are loading much faster. Let me relaunch the app and show you what else we can do. When I tap on an image, the app loads the full-resolution picture and shows how far away this photo was taken from me. There's also a heart icon at the top right that allows you to favorite that specific picture. To do so, I need an account. I allow people to use the app and browse the pictures without an account, but to save favorited images, sync them between devices, and to upload new pictures, you need an account. So let me log in here.

Great. Let me favorite another picture. Oh, this dog looks cute! Let me add it to my favorites. Wait, why do I have to log in again if I just did? This isn't right. My app should remember my log-in. This worked before. I'm gonna dismiss the log-in screen, as I don't want to log in again.

I previously recorded a trace file after reproducing the issue. Let me open it with instruments to analyze the recording. On the left, there's the task that corresponds to when I pressed the favorite button for the first time.

To the right of it, there's the task that was issued after I returned to the latest tab, and the stream of images were refreshed.

Then, there's the task to load the full-resolution image after I tapped on another dog picture.

And to the far right, there's the task corresponding to the second time I tapped the favorite button.

The first task interval actually contains two transactions.

The first transaction received a 401-status code. This was expected since we were not logged in. The transaction is drawn in orange to indicate that this is not a success on the HTTP level.

Then, there's a large, empty area in the task, which represents the time I spent entering the user name and password.

As soon as I'm done entering these credentials, we retry the transaction. The green color of the interval and the 201-status code indicates it succeeded this time. This interaction of an authentication challenge, entering a password, and retrying the transaction is another case the URL Loading system handles for us, so these two transactions belong to the same task object.

Zooming out, we find the second attempt at favoriting an image on the right. The task object is displayed in gray, as my dismissing of the log-in screen caused the task to be canceled, which is also visible in its label. The transaction interval is displayed in orange, as we got a 401 response from the server again. This task occurred after I attempted to like another dog picture and was prompted with the log-in for a second time. We use a very basic log-in system, where the user sends their credentials the first time, but once the server verified the user credentials, it sets a cookie, identifying the user, such that no credentials need to be provided on following requests. So I would've expected this task to have sent the proper cookie. Let's determine whether that happened. As Kacper explained earlier, there should be a small cookie icon here next to the HTTP method, had this transaction sent a Cookie header. But there's no such icon here, which means no cookie was sent. So that part isn't working. Now the question is, did the server not provide us with a cookie, or is the client not sending one, even though it got one? To find out, we need to investigate the previous transaction, and check whether we got a cookie from the server. Here is the previous transaction, the successful one from the first log-in request. This one does have a cookie icon in the response portion of the transaction label, so the server did send a cookie. That's interesting. So why didn't we send the cookie in the next transaction? To get more information about this transaction and investigate the cookie in detail, I will switch to the "Transactions" list in the detail view at the bottom.

The transaction is already selected here, since the time cursor is placed inside of it in the track view.

The extended detail view in the bottom right shows all request and response headers of the currently-selected transaction.

And here is the Set-Cookie header that we expect. At first glance, this cookie seems fine. But oh, wait, do you see the expiry date? It's March 2020. That's in the past! So the server did send a cookie, but it's an expired cookie. No one likes expired cookies! This will lead the URLSession to not send the cookie, as it will only send cookies that are still valid.

This is a server-side bug. I could send the trace file over to our server folks for them to investigate the issue and have it resolved. Now that we fixed the cookie issue, I can favorite a couple more pictures without being prompted to log in. In addition to the "Latest" tab, there's also a "Favorite" tab, where we display a list of all the dog images that the user has favorited. Let's switch to that tab.

Great, there are a few favorites here that I added yesterday, but for some reason, my recent favorites aren't showing up. Let's try again. Let's pick this dog, who seems to be enjoying a bath, and let me favorite it. Let's go back to my favorites and check if it appears.

Hm, it's still not there. Let's use Instruments again to figure out what's going on. I prepared a trace file for this already. I expect to find a task loading the list of favorites in the track view, but it's not visible at first glance. Let me choose the track for my server domain, to display only the requests issued to that domain. We could then go to the detail view at the bottom, which contains a list of all tasks for this domain.

There's quite a few requests here. Let me use the detail filter at the bottom left to search for all requests related to "Favorites," so I can verify whether we even made a request.

Upon filtering, the results show we sent several requests to load the list of favorites here. Let's focus on the track view.

The cursor got positioned at the start of the task I selected down in the detail view, so that makes it fairly easy to find it in the track view above. Let's zoom in to double-check.

So this was the first time we loaded the list of favorites on the initial app launch. This is fine.

Here, I favorited a new image, and after that, we loaded the favorites again.

Well, there's a task interval here, but it's very short.

Yeah, this GET request only took a couple of milliseconds. That's too fast to get a server response. Let's switch to the "HTTP transactions by connection" view again to get more details.

The first thing we notice is that this transaction is not executed on a Connection, but on "Local Cache." This shows us that the request was never sent on the network, but rather loaded from the local cache. This also explains why there is no "Waiting for Response" state, since the transaction did not wait for a server.

So that's the problem: our request is cached, so we don't actually ask the server, and always get the cached response back. One way to fix this would be to tell the server to set a cache-control header, to never cache this response. What we want is to reload the images every time the user goes to the favorites tab and new images have been added. What we don't want is to load the whole list of images if there was no such change. A good trade-off would be if we could ask the server, "Hey, did anything change? If so, please let me know." That's actually something we can do by setting a cache-policy on the request.

To update the code, let me go back to the task view and select the task in question.

For each URLSession Task that got executed here...

We display the backtrace on the right, where "resume" was called on the task.

It was resumed in the method sync, in the ImageCollection type. Let me open this in Xcode to make the change here.

Here, I have my URLRequest, and now I wanna set my cache policy.

The cache policy I want is reloadRevalidatingCacheData, which means that we ignore the local cache and will make a request to the server to check whether our cache is still valid. If so, the server will send a 304 response code to let us know to use the local cache. If not, it will send the new data back. Let's give it a try.

So these are my current favorited images, and the dog taking a bath has been added. Let's add another one.

Now, let's check the "Favorite" tab. The image I just favorited now properly appears. OK, great! That's fixed now as well. Back to my colleague Kacper to cover checking that your app and dependencies behave like you'd expect. Just like Sergio showed before, when I click on the "Favorite" tab without being logged in, the log-in view is presented. We already added Sign In with Apple to make the log-in experience seamless. However, our company has several pet-themed apps, and another team is working on a shared log-in SDK to allow users to reuse their account between the applications. This SDK is currently in development, and the other team has asked us whether it could replace our classic log-in screen. I got the SDK binary, called Pets, which is distributed as an xcframework so that it can be used on all platforms. Integrating it into my Xcode project is as easy as dragging and dropping it in the embedded frameworks section. Now, all that is left is to add a button to our existing view. I will navigate to the source code of our Log-inView.

I will first import the framework, and then add the button to our SwiftUI VStack, just below the Sign In with Apple.

Let's refresh our Swift UI Preview.

Here it is. "Sign In with Pets" button appeared on the preview, exactly where I want it. That was, indeed, a really easy integration. I am curious to check how quick this new log-in method will be. And to measure this, I'll profile my application with Instruments by using Product Profile Action.

I am choosing Network template. And clicking "record" button in the toolbar to launch the app.

My app has now launched. I can now switch to the Log-in View. Instruments is showing all of the networking traffic occurring in the meantime. I will expand it to inspect my app's URL session.

Here it is. But wait. I would expect only my main app URLSession to be here, but seems that the Pets framework we just integrated is making requests from its own session, without me even clicking on the log-in button. That's unexpected. Let's stop the recording right now to investigate it further.

I will zoom in to a few first requests, using option-click and drag.

There's many requests to some analytics endpoint, and to get more details, I can click on this "Pets Sign On Network" session and list all of them in the detail view.

All of them are POST requests, and when I click on one, I can see the backtrace on the right that tells us which part of the code the request originated from.

So seems that request is going through CFNetwork, invoked by Pets, just as expected. But when we navigate deeper, it seems like CoreLocation is being involved. That's really suspicious, especially because I didn't perform any action to trigger it. I wonder if my location is being sent back to the server and that's why CoreLocation and CFNetwork are in the same backtrace.

I will verify that by inspecting the corresponding HTTP transactions for these tasks. To do this, I will switch detail from the list of tasks to the list of transactions. And select one of them.

In the extended detail on the bottom right, it's visible that this request contains some pretty standard headers, nothing to worry about. But wait, look at the request body. It's including my location coordinates, and that's really bad. Sending this information violates users' privacy. We don't want to gather their location without their consent and without a good reason. So far, our app only requests this permission for legitimate purposes that make the user experience better. At this point, I will not go any further with this SDK integration. Instead, I will file a bug report on the other team to inform them about this unacceptable behavior that I detected. And I can even use this Instruments trace to generate necessary information for the bug report. Let's save it on my desktop first.

I will name it "PrivacyViolation" and hit "save." xctrace, command line tool bundled with Instruments, can be used to export this trace to the HTTP Archive format, which is an industry standard for exchange of information about HTTP Traffic. To do this, I can simply run xctrace export command, with input of my trace, and HAR export flag. Let's run it now.

This command generates a file that I can now attach in the bug report. Someone receiving it can inspect the recorded information in any tool that supports HAR, even if they don't have Instruments installed on their machine.

HAR itself is a JSON-based format, so it can also be opened in the text editor or easily processed using scripts. And even though it doesn't contain instruments-specific details, like URLSessions or backtraces, that still should be enough for the other team to investigate this issue.

And that's how you can use HTTP Traffic Instrument to diagnose source and content of traffic coming from your application to make sure that you are in control of what your app does at runtime. Now that you're familiar with using the new HTTP Traffic Instrument, go ahead and target your apps to detect problems just like the ones we showed you today. For easier debugging and having more context while doing so, name your URLSession and task objects.

Always aim for adopting latest networking protocols. And even if you don't find any performance or correctness issues with your app, go ahead and verify by how much data you're sending to get rid of any unnecessary traffic. Thank you for watching today, and we hope you have a great time tracing your app's HTTP traffic. [upbeat music]