Designing for Adverse Network and Temperature Conditions

Good evening, hello.

Welcome to Designing for Adverse Network and Temperature Conditions.

Whether you're just starting out with your first app or if you're a seasoned app developer, we want you to design for a world class experience.

Your apps have the potential to be used by millions of people in so many different situations.

Many of those people will not be on a superfast 4G network, and others may be in warmer environments.

Now, we all do some level of testing, but are you doing enough to know what it's like to interact with your app the way those users would, and if you are, are you providing the best experience that you can? Great apps continue to work well even under challenging real-world situations, which might be difficult to design for. That's why we're here to share some tips with you and brand-new tools in Xcode to help with that. I'm Alex Kara. And I'm Ilya Veygman. We work in iOS system performance so that the system works well and reliably and consistently in the real world. IOS reacts to changing network and temperature conditions, and we want you to be able to do the same thing with your app so that you just can experience your app the way that you designed it.

We have some exciting things to talk about today. First, we're going to dive into real-world device conditions and where they fit in to your design process.

Next, we'll show you how to improve your app's behavior under different network links with new and existing developer tools.

And finally, we'll reveal a brand-new way to optimize your app with varying temperature conditions.

I want you to picture how and where you use iOS. You don't simply use your devices at home or at the office.

You take them with you everywhere. To the beach, to the park, on the subway, in your car on a long road trip.

These might be places with a lot of sunlight or heat or with weak network connectivity.

Now, picture your users. Chances are, they'll be using your app in environments just like these. It's important for you to consider how that compares with your development and test environment.

Now, you're probably doing most or all of your development and testing in your office or in a lab. These are all certainly going to have fast, reliable internet connectivity and climate control, and that's a good thing. We all want a good controlled place to work.

But these won't be the same conditions your users around the world face when interacting with your app. This difference can be a big reason why you might be seeing complaints about your app but dismissing them as one-offs or corner cases. In addition to this, these devices can multitask.

Your users might be in the passenger seat of a car, getting turn-by-turn directions while streaming music even wirelessly to CarPlay.

They might be at a coffee shop charging their iPhone and using it as a hotspot for their Mac. Or, they might be using ARKit to recognize objects in your app through the camera while the app runs 3D rendering or some other kind of complicated background processing. The point of all this is all of these scenarios can cause your device to work harder and get warm.

While your app's features may work well in your test weaves under isolation, are you taking into account how the performance of your app might be different during these very real and common use cases? A potential pain point for your users is when they try to interact with an app designed only for the conditions you've been working in.

How might that look like to them? Well, we noticed some reviews on the App Store which mentioned worsened behavior in certain situations. Great apps, which would sometimes feel short. Whether it's on the train, places with a bit of heat, or on a road trip, this might end up being the biggest way your users remember your app, and they might not want to come back to it after an egregious experience. Now, we know that people use their phones in direct sunlight and that people go into tunnels and expect your app to continue to work, and we know that people aren't always in the best network situations.

It's easy to overlook this when your development conditions are less representative or clean room.

So we want to take these conditions into account, and we want you to provide a consistent experience, not one that's worse than it needs to be. You'll find users connected to 3G networks in your feedback, or you might find a problem in your app and notice that your device was warmer to the touch. Now you might be tempted to think of these as expected bad behavior, but these are not edge cases. These are real situations that you and your users will face. To be able to handle them better, you're going to need the right developer tools and the process that lets you apply them well. So, let's get started with network link. If you're using the network either for primary functions in your app or for background work, you might have made decisions in your code to act on the type of network that you're on.

You might be choosing to time out if network calls take long.

Now, for your users, some of them might be on 3G networks.

If that's their reality, they might be happy to wait for a download to finish even if it takes longer.

But your decision to timeout will not respect their wishes, and they'll be surprised when the app makes no progress even when they're happy to wait. It's these decisions that add up to become part of your user's experience.

When users launch your app, they don't want to see a network spinner that looks like it's perpetually downloading or one that totally stops.

This might be the experience you provide if you're doing network calls during your app launch, but it might look fine to you if you ran this on LTE or fast Wi-Fi even if you were doing performance testing. After some time, your users might give up on this screen and maybe even on your app entirely, especially if there are other apps performing under the same conditions. Your app should continue to be launchable even without a network connection. And if I'm trying to catch on important news with an app that times out early, I might notice that content fails to load on a slower network connection. Now, if I use other apps and notice that they don't fail in the same way, I would think that this app is frozen or that it's buggy.

This situation could escape your development or testing if you're mocking out network calls or explicitly skipping them in your tests.

So, you should go and take a look at Xcode's Scheme Editor and see if you're setting an environment variable for unit testing purposes.

When you're running your app as a unit test host, you might be using this to prevent unneeded work happening during your app's launch. For unit tests, it's okay to skip work such as kicking off background network requests in order to optimize for execution speed, but you need to make sure you're still covering those cases elsewhere.

XC tests will wait until your app delegates did finish launching method returns before it begins running tests. If you're using an environment variable here, do check that the code you skip is truly nonessential for your unit tests to run properly.

If you're mocking out or skipping network calls entirely, you need to make sure you're covering those cases along with realistic network types somewhere else in the development process, and to do that, we need to consider what a good testing model looks like. Earlier this week in the testing and Xcode session, we heard about the Pyramid model as a guide for how to structure a maintainable automation test suite.

A good testing model balances thoroughness, quality, and execution speed and consists of a large number of focused unit tests. These are where it's okay to optimize for fast execution times, and because we want to isolate features here, it's okay to run these in clean room conditions.

You might be using these to look for functional regressions.

These are complemented by a smaller number of integration tests targeting a discrete set of classes in your app. Integration tests check that your app's subsystems work together from the perspective of a user.

Since these test results will more closely reflect real-world use, they may come with an increase variance. So you should be prepared to triage the failure reasons in more depth and not just consider these tests flaky. And finally, the suite is topped off with user interface tests or UI tests, which exercise the app in a way very similar to how users would interact with it. This is your place to validate that all the pieces of your app are hooked up and interact correctly with external resources like the network. So this is where your most representative tests might be.

At the same time, this is where you might see the highest variance in your results, and because of that, it might feel tempting to place more focus on unit tests, and that might lead you into a false sense of security about your app's behavior.

When applied well, this testing model can provide a comprehensive picture of how your app's code base is functioning.

This is great for the testing coverage enables you to do, but you need to be mindful of the testing you might be leaving out in your integration and UI suites.

Putting your focus entirely on unit testing comes with clean room conditions, and while that helps you find regressions, you might be leaving out real-world behavior that you can make progressions under for your users. Now, it's easy to fall into the habit of clean room testing because it gives us many of the qualities we like to see, repeatable results, low variance. These translate into reduced test flakiness.

Now we want your functionality and performance wins to translate into the real world, so you're going to need the right developer tools that also have these qualities.

Variance can be a consequence of realism, and it deserves the same attention you would give to testing and triaging those tough edge case in your source.

As you apply the Pyramid model to different parts of your development workflow, you'll find the right places to create quality checkpoints that work for you and your team, like making sure that all unit tests pass before you merge, so you can find regressions early.

Now, while integration and UI tests might not fit in well as an early checkpoint because of the variance that's incurred when you bring in real-world conditions, they must still have a place in your process. If you make sure you're running them at appropriate times, you'll be able to characterize your app's behavior and find areas of improvement and behavioral progressions that you can make.

So, now that we've made room for real-world conditions, you can bring back and focus on the testing you might have skipped earlier, like those network ones.

So, we've seen some approaches to do this like using custom routers that condition the network infrastructure, and this can be really successful when done right.

But this can also be really difficult to do, especially if you're a developer just starting out.

Even with a good testing model and a strong focus on triage, to conduct real-world testing, you'll need good and reliable developer tools.

So, if you're targeting macOS with your app, you should download and use the Network Link Conditioner Preference pane. You can use it to vary the network type and see how your app behaves under networks like 3G or EDGE.

The Network Link Conditioner is available on iOS from the developer settings menu on devices that you're using for development. From here, you can vary the network type between contended or more representative presets and design your app for them without the need to set up or change the network infrastructure.

This is a reliable and repeatable device supported way to run your app under different networks.

And if you have custom needs, you can also create customized presets for specific types of bandwidth, packet loss, and latency that you would like to design for.

This is great to check how your app behaves in specific environments. In Xcode 11, we've brought the ability to activate and vary different network types to the devices and simulators window so that you can start easily and reliably including realism in your design process. You'll see a new device condition section on the lower part of the window. From here, you can put your device into a more representative state.

If you want a network link, you'll see all of the network types from earlier as well as new profiles to vary the network quality itself.

This means you can have your device and your app behave as if it's running on networks like 2G or EDGE, 3G or LTE, or different types of Wi-Fi. You can even choose the quality of the network type like a good EDGE network or an average 3G one. Now, people do use connections like these, so I hope you find it helpful to see how your apps work with them too and look for where you can find behavioral progressions.

Once you've picked a condition you want to activate, click start from the devices window.

Now, these conditions are system wide, so you can expect everything to start reacting differently, including your app.

On devices which have conditions active, you'll see a new gray status indicator.

Although activating a network type affects the whole system, the UI indicators for your network will remain unchanged.

You should also know that an activated network condition is a ceiling or a cap on your network type, and it cannot upgrade your network performance from where it actually is in the real world.

From the device, if you tap the gray status icon, you'll see a prompt telling you about the active condition as well as an option to stop it, and if your device is disconnected from Xcode, the condition is automatically stopped.

To show you how you can use network link device conditions to find areas of progression in your app, I'm going to bring Ilya back on stage. We often expect our apps to behave slightly worse on a slightly worse network link. But it's important for you to be asking yourselves whether the behavior is as bad as it really needs to be. Could it be better? Are there progression we could make under an adverse or different network.

Here's a mind-blowing example app. We can launch this to look at baseline behavior for a network connection with an ideal lab condition.

This app probes an endpoint we set up just for this demo to see how long it takes to make the connection. We see that on average this connection takes around 150 milliseconds. We can think of this as analogous to something like requiring secure login or streaming content from a website. And this looks great. If we're doing UI testing in our lab, we would assume that everything is working smoothly, and we have no problems.

Now let's see what happens if we turn on the Network Link Conditioner from the Xcode devices window.

This is an average 3G network connection in this example.

Let's see what happens now.

When we run the probe again, we see it takes longer, in this case, on average around a little over 750 milliseconds.

This might not be all that surprising because after all a 3G network is slower when compared to something like LTE or Wi-Fi.

But the important thing to note is as we said before this is the actual network many of your users will see. What can we do to improve this experience for them? So you probably noticed just above the run probe button we have these two switches which are disabled for Optimistic DNS and TLS 1.3. Let's turn them on and see what happens. And now as we turn them on and run the probe again, we see an immediate improvement, around 33 percent faster. Simply by testing this app with the Network Link Conditioner active, we explicitly noted that there is a significant performance loss when we have a slower network like 3G or when compared to a faster one like Wi-Fi or LTE.

This told us that we should take into account these new features and use them to proactively improve performance even under realistic network conditions.

By running Network Link Conditioner, you'll notice some behaviors are worse than they need to be whereas previously you might not have. Here are just a few things you can do to proactively improve the overall experience.

First, do set reasonable timeouts.

That is, timeout when you stop making progress, not just when progress takes too long. As we said before, if your users are at 3G network, you might be happy to wait longer for content to load. An arbitrary timeout will be a worse user experience for them. Additionally, do use HTTP/2 and do avoid reachability checks whenever possible. Instead, just try to use a network and do what you can to make sure your app works well under as many network conditions as possible. To learn more about what else you can do, please see these two sessions from last year's WWDC as well as Advances in Networking parts one and two from earlier this week.

So, get conditioning. You want to start considering real-world network usage in your app. You want to use network link device conditions to characterize your app's behavior under that usage and ask yourselves, is this acceptable performance or can it be better. We recommend that you test with at least 3G networks and look for the progressions that you can make. You want to vary both the network type and its quality to see if you're still providing a good experience and then you can lock in those performance wins by making this a part of your integration and UI test runs. Now, I want to talk about varying temperatures.

People like to go outdoors and use their devices on a sunny day. They might head to coffee shops and use personal hotspot while iPhone is wirelessly charging.

In these situations, devices will start to feel warmer, and that's normal behavior. Some thermal conditions can cause iOS devices to change their behavior or performance in order to regulate their temperature. And temperature can vary for any number of reasons, whether it's an increase in work done by the device or environmental effects like exposure to direct sunlight and many more.

All of these are normal scenarios, and iOS subsystems react to changing temperatures to regulate the impact of those effects. But what's missing is for your app to work well under changing temperatures and how you react to them too.

Now, when certain thresholds are exceeded, for example, if the device is left in a hot car for a prolonged time, users might see this temperature warning screen.

At this point, they can no longer interact with your apps.

Part of the reason this happens is to provide users with the crucial ability to make emergency calls should they need to do so.

Now, the system is doing what it can to limit its energy impact, which effects heat and battery life, and your app is a resident of the system as well, and it's important that you take into account its energy impact too.

To do this, you can start changing your app's behavior dynamically when you're in a different thermal state. By designing defensively, you can reduce your app's energy impact by turning off background work, which contributes to higher thermal states.

You can register for thermal state change notifications and look up which state the device reports to your app and consider scenarios that are normal like device warming, because the system knows how it should react to an increase in temperature.

But your app knows more details about the work it's doing and how that work should react to a higher thermal state while maintaining a good experience.

So, let's take a look at these thermal states that you might see reported.

At the nominal state, the device is at normal operating temperatures, and there's no need for any corrective action from your app. At the first state, we recommend that you proactively start some energy saving measures so that you're not contributing significantly to an overall increase.

When iOS sees thermal state fair, we start to pause discretionary background work like photos analysis. When the device reports a serious thermal state, system performance will be impacted, and your app should start stronger energy saving measures and reduce heavy CPU usage, graphics, and I/O. At this point, you should use lower quality visual effects. Some measures we take on the system include lowering the frame rate of ARKit apps and Facetime so that they're less intensive. And if a user is restoring from an iCloud backup, they'll find that it will be paused at this state until the device cools down.

And at thermal state critical, your app should stop using peripherals such as the camera. If you're ending up on the top of the list on the battery impact screen, users might even consider deleting your app. Together with the system, your app should dynamically react to these changes so that you can continue to maintain a good experience while keeping your energy impact low.

To learn more about the state cases and our recommendations, you can take a look at our documentation. And Ilya is now going to show you how you can dynamically react to these states in an example. I'm going to show you a sample ARKit app based on a modified version of our existing sample code handling 3D interaction and UI controls in augmented reality. I took it for a stroll in Apple Park and it's performing some heavy background work too. Here you'll see the app running under nominal conditions. You can see the red focus square turn solid, finding a surface, and letting me drop a nice chair and a lamp for myself to sit in and do some reading.

Now you can see the camera movement is quite smooth. Everything stays in place. Everything stays in place. Everything is behaving just like it should be. Now, let's look at this same app again, but now I've been outside for a long time. I've been sitting in the sun. It's warm out, and the device has warmed up.

You'll notice two things. First, the frame rate is not quite as good as before, and second is that despite the fact that I'm aiming almost straight at the ground, the focus square does not have time to find a surface. This would not be a great experience for your users, and it might be a little frustrating for them. So, what can we do about this? First, you should register for the ProcessInfo.theremalStateDid ChangeNotification.

When you receive a thermal state change notification, read the actual thermal state and then react accordingly. Depending on your state, you should enable or disable certain features to ensure smooth functionality or whatever metric you find is important. Here's an example of how you can register for the thermal state and then read the thermal state. And here is how you might choose to react to the thermal state.

In this scenario, under nominal and fair, I have all my features enabled. In this example, I have face tracking, person segmentation, and motion blur all turned on. As the thermal state increases to serious, I disable face tracking and frame semantics, but I leave motion blur on. And at critical, I turn off everything.

Now that we actually react to thermal states, let's see how this app behaves again in the same scenario where we've been outside for a long time, and we see it's much better now. The focus square finds a surface. I can drop my chair and my lamp just like before, and I can get some reading done. Now, it's good to code defensively and to react to thermal state changes, but you want to know ahead of time if this works as you expect.

In general, we can do better by testing how we expect our app to behave under varying temperatures ahead of time. In other words, you should test your defenses.

But for something like this, how do we even go about that? Thanks Ilya. But the thing is, not all of us are going to have access to thermal imaging.

So, just like with network conditions, we recognize the challenge in trying to verify your app's behavior and that there's high variance in existing approaches.

We've noticed some methods that people are taking which we would not recommend like running a dummy CPU load to warm the device, throwing away the first hour of results, and then profiling app behavior when the device is hot. So, we've been hard at work to provide a developer tool for this, and we came up with a way to reliably change the reported thermal state on the device without physically warming it and still keeping it safe to use. And we built that way into device conditions in Xcode 11.

From the same devices and simulators window, you can activate elevated temperature conditions and have your device reach different thermal states without it needing to be physically warm.

So now, you can quickly and easily have your device report the fair state, to test your proactive energy-saving measures.

Thermal state serious, to check that you're reliably lowering your resource usage and energy impact, and thermal state critical, to see that your use of peripherals do actually stop.

Running one of these causes the device to behave exactly as if it were really in that thermal state, but before you start using these, there's more you need to know about how they work on your device. Ilya. Thanks Alex. I'm going to show you more about how this condition works under the hood. You can see a graph here representing the actual thermal state of the device, the active condition if there is any, and how warm the device actually feels to the touch, also represented by a thermometer at the top right. Imagine a baseline device on your desk. It's at room temperature. You have no condition active, and you haven't been using it for a while. Here, the thermal state is at nominal. If you now activate the serious thermal profile, the device will ramp up over time to serious starting from nominal, reaching fair, and eventually reaching serious. This process will take a few seconds. Just like in real life, if you are subscribed to thermal state notifications, you'll receive a notification at fair and at serious. Now, there are two important things to note about this.

The first is that your devices has not actually warmed up or changed in temperature as a result. The second is that this does not fix your thermal state, but it rather acts like a floor. Let me explain what that means.

Imagine you have your device in this condition, and you ran some heavy computational load, or you just left it out in the sun for a while.

And now the underlying temperature is actually increased. The device feels warm to the touch.

Regardless of why, the thermal state will actually also increase from serious to critical.

This is a precaution to ensure your system still behaves safely even if you do very heavy testing under the thermal condition. If you then stop using your device or let it cool down, the thermal state will return back down to serious and remain there until you tear down the condition, at which point, the device will ramp from serious to fair and down to nominal.

In all these situations, you will receive thermal state user notifications. In Xcode 11, this thermal state information is visible in an energy gauge in the debugging navigator.

There are two thermal state tracks here, both located toward the bottom of the energy impact section. The bottommost track shows the actual thermal state of the device, color coded for easy interpretation.

Here, you can see the thermal state ramping up and then down in reaction to the active condition. You can see in this scenario it took around 10 seconds in each direction.

The top track shows the active thermal device condition, if there is one.

To show you more about debugging and optimizing with Xcode and what tools you have at your disposal, I'm going to call Jay on stage.

Hi everyone. I'm Jay. I'm part of the Energy TechnologyTteam in Core OS, and I'm here to show you how an app behaves when the device is thermally constrained and what can you do about it.

For the purpose of the demo, we're going to use a modified version of the Fox 2 app that's a publicly released sample for SceneKit from a few years back.

Let's get started. I have a device that's running the app without any thermal condition active. Let's see how it loads.

This is how the app looks.

First, let's look at the bottom left corner of the screen.

That's the FPS. We can look the FPS is consistently hitting more than 30. Let's look at the details on the app.

There's a nice vignetting around the app for a cinematic effect. Let's look at all the details inside the app. There's just tons of details.

If you look at the green gem on the right side, there's a light source on top of it, and it's doing a great shadow.

If you look at the moving objects, they have light sources coming out of them, and they're doing great shadows on the fox as well. Let's look at the lava. There's smoke coming out of it, and the GPU is doing a great job of blending in with the background.

There's a lot of tiny fire particles coming out as well. This is a great user experience.

Users really like to use apps like these.

If you're running performance tests, they're going to be all green. Let's switch gears and see what happens when you run a thermal condition. I have another device running the same app but with a serious thermal condition active, and let's see what happens. I'm going to switch over to the device. Now, if you look at the bottom left corner of the screen, the FPS is down to 17. We lost almost half of our performance. If you look at the moving objects, there aren't as buttery smooth as they were before. If you look at the moving platforms or the moving rocks, they aren't as smooth as they were before. What can we do to fix this? We went ahead and we modified the app to listen for changes in thermal state.

Whenever the thermal state changes, the app is going to respond dynamically and reduce the features it supports. Let's see how that works. I have a small debug UI on the top that can switch it from being static or dynamic, and I'm going to switch it to be dynamic now.

If you look at the bottom screen now, we're close to 20 FPS again. The scene looks similar, but we got rid of some of the details. We got rid of the smoke coming out of the lava. We reduced the fire particles a bit, but you can see, the response of the app is still great.

That's what we're looking for. Now, let's look at the code changes we had to do to make this happen.

When you're at nominal or fair, we don't have to do anything. You can enable all of the features on the app. We have HDR. We have depth of field. We have soft shadows, and we also have the post processing set to high.

As soon as you hit serious, we start doing something. We react by disabling HDR. We also turn the shadows from soft to blob. We also set the post-processing to medium. When we are at critical, we go even further. Critical is a very high thermal state, and we disable as many features as we can. We disabled HDR. We disabled depth of field. We disabled shadows, and we also turned on post processing.

All of this is going to help to keep the app responsive at all times. Now, let's look at some tools that we can use for tuning for thermals.

This is an instrument stack taken on the same app with and without any optimizations. Both have been captured with a serious thermal condition active. Let's look at the FPS track. This is the time display was shown in the same frame. Let's decode it a bit. When the display is showing one frame, the GP is working on rendering the next one, and the CP is working on creating the instructions for the one coming even after that. What happens is when, and without any optimizations what happens is that the GP is not able to deliver frames in time, and the display keeps showing the same frame.

This is what a stutter looks like. This is when your app is lagging. If you notice, after the optimizations, the frames are spaced consistently. Apart from using instruments, you should also be using the Xcode energy gauge. You should focus your attention to the average energy impact your app has.

The higher the energy impact, the higher the battery drain and the higher the chances of your app causing a rise in the thermal state. If you look at without optimizations, we had a very high energy impact, but with the optimizations enabled, we were able to lower the energy impact. What that means is the app is not going to contribute to a rise in the thermal state when the device is running the app.

Back to Alex and Ilya for a recap.

Thank you, Jay.

If you want to learn more about debugging in Xcode as well as other things you can do to bring real-world scenarios to your development process, such as environment overrides, please take a look at this session from earlier this week.

So, we heard about real-world conditions that people will be using your apps under, like 3G networks or elevated temperature states. And we heard about the importance of providing the best experience truly possible in those states.

We heard about how a typical development and testing workflow might natural steer you towards clean room in an attempt to avoid flaky tests and variance. And we heard about the new device conditions in Xcode 11, which lets to you quickly and easily put your test devices in adverse network or temperature states. That means no more waiting an hour for devices to actually warm up and no more throwing away test results. This is a great way to ensure the code that you're designing and all the great performance and features that come with it translate into progression for your users in the real world.

So, to recap, do use the Test Pyramid model for organizing your automation test suites and be prepared to triage your results as you introduce real-world conditions to your testing. Only skip truly unnecessary code in your unit testing. And there's a call to action here.

Don't forget to take your conditioners with you. Do activate a device condition, see how your app behaves, and add test runs to find that egregious behavior that you might have missed.

We recommend, again, that you do test with at least the 3G network type of varying quality, see how your app behaves in the serious thermal state too.

We are really excited to see what progressions you can make with the device conditions we've made available. And we'd love to hear from you. Contact developer support or come to the Xcode labs right after this session. And for more information, see our session link. Thank you so much and we hope you have a great WWDC. [ Applause ]