Technical Note TN2431

App Testing Guide

This guide covers recommended procedures for app testing. Though the importance of testing apps and app updates before submission or deploying to Enterprise users is common knowledge, this guide highlights specific areas that are frequently overlooked. Additionally, strategies are provided to debug general problems that might occur in your app's distribution build.

Introduction
Terms definition
App Testing Procedure
Common causes of customer facing issues
Release build debugging strategies
Persistent Issues
Related material
Document Revision History

Introduction

Before submitting your app or its update to the App Store or deploying to Enterprise users, apps must be tested in a simulated customer environment in order to guard against failures that result from untested circumstances. This guide covers the general process to do so including common areas that new apps and app updates frequently miss during testing.

Terms definition

This document uses the following terms:

Figure 1  A problem that only occurs with the Release build can surface in the varying environments pictured here.

App Testing Procedure

The recommended app testing procedure follows. Because there are differences between the app's Debug build (ran during development) and the Release build (the app Xcode optimizes for submission), the full app testing procedure covered by this guide should be used to maximize the chances of catching problems that could otherwise surface in the customer environment.

Reproducing release build issues locally

Should a customer facing issue be reported in your app's Release build, use the app testing procedure to debug the issue in your local development environment. This section also focuses on increasing chances of reproducing a customer facing issue as that is key to confirming the problem is solved after app changes are made to address an issue.

1. Test your Release Build

Release build testing involves a unique workflow in Xcode that is different from Building and Running connected to the debugger. The difference is important because they can, when properly facilitated, reveal problems that weren't visible during development. The largest source of issues found in the customer environment are due to lack of thoroughly testing the Release build, and respecting the large number of fundamental differences between development and distribution builds.

Testing your Release build can be done in the following ways varying by platform:

  • iOS, tvOS, and watchOS, use one of:

    • TestFlight (this is the preferred method).

      App Distribution Guide > Distributing Your App Using TestFlight (iOS, tvOS, watchOS).

    • Ad Hoc distribution

      App Distribution Guide > Exporting Your App for Testing (iOS, tvOS, watchOS) (Ad Hoc).

  • macOS: App Distribution Guide > Exporting Your App for Testing (Mac).

2. Test on supported OS versions and device types

It is possible that a particular app issue only reproduces on a specific version of the OS or device type. Therefore, testing all the OS versions and device types your app supports is crucial to ensuring a good customer experience.

  • Issues reported on a specific OS version and device type

    If an issue is reported against your app's release build, pay close attention to the OS version and device type it occurred on when trying to reproduce the issue in your local development environment.

  • Maintaining a robust suite of test devices

    It can be difficult to acquire and maintain configuration of all the combinations of device type and OS versions your app supports, but this section helps offer a reasonable strategy to maximize your ability to do so.

  • General recommendations for testing specific devices and OS versions

    • Try to maintain a spread of test devices (32- vs 64-bit, memory sizes, processor speed, GPU, iPad vs compact devices like iPhone and iPod touch, OS releases).

    • Maximize your deployment target (and thus minimize the spread of OS releases you need to cover).

    • Buy devices on a regular basis; iPod touch is your friend.

    • When a new OS release is imminent, carefully plan which devices you’re going to take to the new OS release and which devices you want left on the old OS release; upgrade the latter before the new OS is released.

  • Obtaining the device type and OS version related to a specific issue

    The following examples illustrate ways to get this important information:

    • For crashing Issues, the OS version and specific device type can be found at the top of a crash log. As an example, see the OS and device in the example crash log shown in Analyzing Crash Reports section of TN2151 - Understanding and Analyzing iOS Application Crash Reports.

      • Crash logs that are generated in App Review will be attached to the rejection letter.

      • Crash logs that happen for your App Store customers or TestFlight testers are available in Xcode; the following guide walks through obtaining these types of crash logs App Distribution Guide > Analyzing Crash Reports.

    • For problems reported by testers or Enterprise users, the OS and device type should be collected by the user in the following ways varying by platform:

      • iOS: Settings app > General > About, "Version" and "Model".

      • tvOS: Settings app > General > About, "tvOS" and "Model".

      • watchOS: Apple Watch App (on the paired iPhone) > My Watch tab > General > About> "Version" and "Model".

      • macOS: Apple menu > About This Mac, and System Information utility > Model Identifier.

3. Test a clean installation

Incidentally creating a dependency on the presence of previously-created data while iteratively developing an app is a common problem. Xcode's app install process is optimized for development, but is slightly different than how iTunes and the App Store install apps. Prior to distributing a new version of your app, thoroughly test launching from a clean installation – one that is free from any data that might have been created by an older version of your app.

  1. Remove previous builds of your application from the testing device.

    • You can delete an application and its container from the iOS home screen or using Xcode's Devices window.

    • tvOS apps can also be removed in Xcode's Devices window.

    • Watch apps can be removed using the paired phone's Apple Watch App.

  2. If your application uses a shared app group container to store common data files, you should also remove all other applications in that app group from the testing device. Shared containers are not deleted until all applications in the app group are removed from the device.

  3. Keychain data previously created by your application may not be deleted when the app is removed from the device. To clear its keychain, an application can call the SecItemDelete API with a query that matches all existing items. You might find it more convenient to prepare a small utility application that you can quickly install on your test device for this task. This utility application must have the same bundle identifier (and App ID prefix) as your real application to modify its keychain.

4. Test an app update

Unless this is the a new application, the majority of your customers will be upgrading from a previous release. Test that upgrading the app on top of a prior installation of your app works smoothly and, if you have made changes to your file format, that existing data is successfully migrated, or continues to be supported in some other fashion. This recommendation generally applies to all platforms, including macOS.

App update testing recommendations specific to watchOS, tvOS and iOS:

  • Save and restore app containers using Xcode's Devices Window.

    • An app's container that is saved through Xcode includes data that was created through use of the following APIs:

      • Files saved through NSFileManager to locations defined by NSPathUtilities, such as NSDocumentDirectory, and NSCachesDirectory.

      • App preferences that are saved through NSUserDefaults.

    • To test an app in a specific state, save the app's container on the device whose state you want to preserve, and restore it on the testing device using the steps in:

      Devices Window Help > Managing Containers on a Device.

  • Follow the general app update testing procedure covered in:

    TN2285 - Testing iOS App Updates.

The issue of forgetting to test the app-update scenario before going live is one of the common customer-facing issues covered in this guide; see section: App update process. If you're experiencing a different customer-facing issue in the Release build, the next section lists other common problems that are frequently first-detected in customer environments.

Common causes of customer facing issues

Sometimes issues that surface only in the customer environment can relate to the build configuration that was used to build the app (Debug versus Release) or, the code signing profile used to sign the app (Developer versus Distribution). Other times, the issue only reproduces when disconnected from the debugger, running on an macOS guest account, or ran within differing network or memory-availability conditions. This section lists common causes of release build differences; often times, solutions or workarounds can be inferred once specific triggers are identified.

Debugger effects

There are a couple monumental differences that result from running your app through Xcode (therefore, attached to the debugger). This section covers the differences that frequently cause problems you should therefore be mindful of.

  • The debugger disables Watchdog timeouts

    The Xcode debugger disables crashes due to watchdog time outs, and for this reason, crashes due to watchdog timeout are often noticed only in the Release build. The easiest way to check for watchdog timeouts is to disconnect the Lightning cable and run the development build from the home screen. For more information on watchdog timeouts, see:

    QA1693 - Synchronous Networking On The Main Thread.

  • The debugger prevents your app from being suspended

    Because the debugger prevents apps from being suspended, your app must be launched from the home screen in order to test any background processes your app might implement. For example, NSURLSession allows an app to opt into a background session, however this code will not be properly tested until the app is run disconnected from the Xcode debugger. For more information on this topic, see: Apple Developer Forums > Topic 42353.

Device power

iOS may behave differently depending on battery charge level, how and when the battery was last fully charged, and whether or not the device is currently being charged. Therefore, to confirm your app behaves properly across the myriad of differences that can result from power level, you should:

  1. Test the app unplugged from a power source.

  2. Test the app in Low Power Mode.

    • Steps to enable Low Power Mode on iOS:

      1. Navigate to Settings > Battery.

      2. Switch on Low Power Mode.

Build configuration

Exercise to determine if an issue is Build Configuration related. What configuration was used in the build that failed?

  1. Set the Run task Build Configuration to the configuration used to build the failing app.

  2. If the issue does not reproduce, the problem is likely not related to the build configuration.

  3. If the issue does reproduce, now set the Run task Build Configuration to the opposite configuration used to build the failing app.

  4. If the issue does not reproduce, this confirms the issue is related to the build configuration. Check each build setting that is different across the configurations for potential causes of the problem.

Notes on Build Setting differences:

In Xcode's Build Settings almost every setting has the capability of being set differently for each build configuration. The default build configurations are Debug and Release, and by default, most build settings are the same. As an example, if Xcode were using a different Info.plist file for release builds, there is a good chance the Info.plist build setting differs across the two build configurations. Because the Info.plist build setting is the same across the two build settings by default, a change to this setting must have been made explicitly by the developer of this project.

Figure 2  Example build setting that shows how something as crucial as the Info.plist could be set differently for Release builds.

Compiler optimizations

A build setting that is different by default across the two build configurations is Optimization Level. This build setting controls compiler optimizations, which are code optimizations that are made by the compiler at build time. Compiler optimizations are on by default for Release builds because they result in a more performant app, however, you should be mindful of differences in behavior that can occur if you do not follow language rules.

For example, in Objective-C, the following article covers a hypothetical example of a Release build difference that results from compiler optimizations:

LLVM PROJECT BLOG - What Every C Programmer Should Know About Undefined Behavior.

For a Swift example, see the optimization note within:

The Swift Programming Language > Declarations > In-Out Parameters:

"As an optimization, when the argument is a value stored at a physical address in memory, the same memory location is used both inside and outside the function body. The optimized behavior is known as call by reference; it satisfies all of the requirements of the copy-in copy-out model while removing the overhead of copying. Write your code using the model given by copy-in copy-out, without depending on the call-by-reference optimization, so that it behaves correctly with or without the optimization."
Figure 3  The default Optimization Level build settings.

Use the following steps to confirm whether compiler optimizations causing the difference in your Release build:

  1. Reproduce the issue by running the app's Release build.

  2. Temporarily set this build setting's Release value equal to the value of Debug (which is None [-O0]).

  3. Then, build and test a new Release build of the app.

  4. If the issue does not reproduce, the issue is related to a compiler optimizations.

Code-signing provisioning profile

  • Exercise to determine if an issue is Provisioning Profile related • Was a distribution or development provisioning profile used to code sign the failing app?

    1. Create an installable .ipa of the app file using the steps in App Distribution Guide > Distributing Your App Using TestFlight (iOS, tvOS, watchOS) and code sign the app with the same App Store profile that was used to sign the problematic app.

    2. When creating the app, use the process in How to test the same build that generated a crash log to ensure you choose the right app archive. If the issue is not crash-related, refer to the Version column of the Archives Organizer to make sure the correct archive is chosen.

    3. If the issue does not reproduce with the newly created TestFlight build, the problem is not related to the code signing provisioning profile.

    4. If the issue does reproduce, now grab the development signed version of this same app as it resides on disk within the .xcarchive file.

      • Control-click on the archive in Xcode's > Window menu > Archives, and choose "Show in Finder"

      • Control-click on the .xcarchive in Finder and choose "Show Package Contents"

      • Find the .app within the ./Products/Applications folder

      • Install this developer-signed version of the same app using Xcode's > Window menu > Devices, with the steps in Devices Organizer Help > Installing Apps on a Device

    5. If the issue does not reproduce with the developer signed version of the app, this confirms the issue is related to provisioning profile. Check the entitlements of each profile for differences that could potentially cause the issue using the steps in TN2415 - Entitlements Troubleshooting > Inspecting a profile's entitlements.

    6. The best way to confirm which entitlements are enabled on a build submitted to the App Store is to view build details in iTunes Connect. For more information, see:

      iTunes Connect Developer Guide > Viewing App Information > Viewing Build Details section.

  • It's possible that one of the provisioning profiles used during the signing of either the development or release build are out of sync with the current state of enabled capabilities. If it's the distribution profile that is out of sync, the issue produced as a result will only be present in the distribution build and not exhibited in the development build.

User privileges

Testing your app on a Guest Account is an essential step to simulate the customer environment for an macOS app. Mac App Review facilitates review with minimally-privileged accounts for the purpose of catching bugs that developers and testers might have missed testing the app on an Admin account. For example, ensuring the guest account login runs your app smoothly ensures there are no privileges problems or missing files or folders that had been preconfigured within the development environment.

App update process

The following are important notes that relate to issues that can occur during the app update process:

  • Building and Running a pending app update through Xcode does not properly test the case of installing an app update on top of an existing app downloaded from the app store.

  • App Review devices typically do not contain an existing installation of your app from the App Store and therefore, the review process does not test the case of updating an existing customer installation.

  • An example issue that could only be caught while testing an app update is if your app has changed a file format and forgets to check for or support existing customer files in the previous format. The app might crash assuming that all files in the users Documents directory are in the new file format.

  • Test your app update using the steps in TN2285 - Testing iOS App Updates

Network conditions

Debug builds are often tested within an isolated network, whereas Release builds run in varying network conditions that your users are exposed to. The result is an issue can appear to be related to the apps Release build, but may in fact relate only to the network that either build runs under.

To alleviate these kinds of issues:

  • Use the Network Link Conditioner

    This tool runs on your iOS device or tvOS device simulating slow or unreliable internet connections. The Network Link Conditioner can be activated using Settings app > Developer > Network Link Conditioner on the testing device.

  • Test in IPv6-only networks

    It's possible for an issue to surface in an IPv6-only or IPv4-only network, and not the other, so it's a good idea to test your app in both. Since IPv4-only networks are still widely in use, it's more common that an app issue relating to DNS64/NAT64 go undetected before submission.

  • Follow the steps in the definitive guide: Networking Overview > Designing for Real-World Networks.

Memory availability

Another important consideration involves understanding there are different amounts of memory available to various devices and a given customer's instantiation of your app. This section recommends you walk through the process of minimizing your app's memory footprint as the best way to ensure your app avoids memory warnings or termination due to memory depletion in the customer environment. In addition, minimizing memory usage using Instruments can sometimes be a quick exercise that can yield surprising performance gains. To do so, follow the steps in:

TN2434 - Minimizing your app's Memory Footprint

Data edge-cases

Apps that rely on customer-supplied data face the challenge of anticipating uncommon situations while processing that data. The wider the range of variations in customer supplied data your app accepts, the more responsibility there is to detect and gracefully handle unsupported cases. These kinds of considerations during testing are referred to here as data edge cases.

The following are example data edge cases:

  • If your app loads user-defined images, test the loading of an excessively large image. Because many images are compressed file formats, loading an image in memory can require significantly more space than is required to store it on disk. If an excessively large image opens normally in your app, does your app implement a size limit at a certain point larger than that?

  • If your app loads user Contacts, how does the app behave with an extremely large amount of Contacts? The extreme case of one or no contacts should also be tested.

Internationalization

To test every language that your app aims to support you must update the Language preference on the device.

In addition, the Region setting changes the device locale which causes dates to be returned, parsed and presented differently. Here are some example reasons why adjusting Region should be a regular part of your app testing process:

  • If you work with dates or times, test with a 12 hour region, a 24 hour region, with a 12 hour region that’s overridden to use 24 hour, and a 24 hour region that’s overridden to use 12 hours

  • If you work with dates or times, test with a non-Gregorian calendar

  • Test with region that uses non-Latin digits

Here's how to set the Language (localization) and Region (locale):

  • iOS – Settings app > General > Language & Region

  • tvOS

    1. Settings > General > Apple TV Language

    2. Settings > General > Region Format

  • macOS – Apple menu > System Preferences > Language & Region

Release build debugging strategies

Problems that surface in the customer environment can be daunting to debug; this section covers the options that are available to you to collect information that is helpful for debugging problems that occur in Release builds. Xcode's debugger does not connect to apps that are code signed with a distribution profile, and therefore, Crash Logs and Console Logging are key to diagnosing a problem surfacing in the customer environment.

Obtaining crash logs

Use the following steps to collect crash logs with Xcode:

  1. Plug the device running the problematic Release build into your Mac using the Lightning cable.

  2. Open Xcode's > Window menu > Devices

  3. Choose the device from the left sidebar.

  4. Click the View Device Logs button in the center pane.

  5. Sort the crash logs in the left-sidebar by date and view the log whose date and time matches the time of failure that you're troubleshooting.

Use the following steps to collect crash logs without Xcode:

  1. Open the Settings app on the iOS device.

  2. Navigate to Privacy > Diagnostics & Usage.

  3. Select Diagnostics & Usage Data.

  4. Locate the log for the crashed app. The logs will be named in the format: <AppName>_<DateTime>_<DeviceName>

  5. Select the desired log. Then, using the text selection UI select the entire text of the log. Once the text is selected, tap Copy.

  6. Paste the copied text to Mail and send to an email address as desired.

Pairs of crash logs and console logs are ideal

There may often be additional information available in the console log that is helpful for debugging crashes. See section Obtaining app logging to collect those from a customer device. Because crash logs include the date/time of a crash, a large amount of console logging can be sifted by paying closer attention to the events that occurred at-or-around the time of the crash. Though it's not always the case, console logging can often reveal additional information about a crash that is not contained within the crash log.

Symbolicating crash logs

Before a crash log is human-readable, it must be symbolicated – the process that replaces hexadecimal values in the stack trace with actual method names that are used in your app's code and within the frameworks.

Steps to symbolicate a crash log:

  • Follow the steps in the Symbolication section of the following guide.

    TN2151 > Understanding and Analyzing iOS Application Crash Reports.

  • Note for apps with bitcode enabled: dSYM files (that are used to symbolicate crash logs) must be downloaded manually. Use the following steps to manually download .dSYMs for bitcode-enabled apps:

    • Open Xcode's > Window menu > Organizer

    • Select your app on the left-sidebar

    • Click Download dSYMs.

How to test the same build that generated a crash log

Testing the exact build that generated a particular crash log can be a valuable exercise, as this is one of the only ways to ensure that source code changes are not affecting your ability to reproduce a Release build issue.

As an example use case:

 A crash-log from a customer's app crash is pulled down by Xcode and you want to create an Ad Hoc testing build from the same app archive that is running on the customer's device. If there are many app archives in Xcode, or the most recent App Store version is newer than the customer install, how to know which archive was used to distribute the desired version of the app?

Follow these steps:

  1. Take note of the app's version at the top of the crash log, e.g.

    Version: 1.0.3 (1.0.3)
  2. Open Xcode's > Window menu > Organizer > Archives tab.

  3. Locate the archive whose version matches that in the crash log, e.g.

    Art/tn2431_archiveVersion.pngArt/tn2431_archiveVersion.png

  4. You can test this build either of the following ways:

    • To test an Ad Hoc or TestFlight version of this archive, use the Export button on the right-pane of the Archives window. The tested build will be a distribution-signed Release build.

    • To test the development-signed version of this Release build:

      • Control-click on an archive and choose Show In Finder.

      • Control-click on the .xcarchive file in Finder and choose Show Package Contents.

      • Find the .app file inside the Products > Applications folder.

      • Install the app using the "+" button on Xcode's Window menu > Devices pane, after selecting the device to install the app onto from the left-sidebar.

        Art/tn2431_installDeveloperSignedApp.pngArt/tn2431_installDeveloperSignedApp.png

Obtaining app logging

Use the following steps to access your app's logging:

  • In macOS Sierra, it is possible to obtain logging for all platforms (iOS, Apple TV, Apple Watch, Mac) using the "Devices" section of the Console app sidebar.

  • For OS X 10.11 and lower:

    • Obtaining console logging for iOS, watchOS and tvOS apps:

      1. Plug the device running the problematic Release build into your Mac using the Lightning cable. For Apple Watch apps, plug in the watch's paired phone.

      2. Open Xcode's > Window menu > Devices

      3. Choose the device from the left sidebar.

      4. Expand the device's console by clicking the triangle on the bottom-left of the window.

      5. You may also click the down-arrow on the bottom-right to save the console logging to disk.

    • Obtaining console logging for macOS apps:

      1. Open the Console app in /Applications/Utilities

      2. Logging from your app is visible in the center pane.

    • Obtaining console logging without Xcode:

Tips to maximize debugging ability

  1. In iOS 10+, use the Unified Logging APIs rather than NSLog, printf, and print.

  2. The following recommendations apply to logging in general:

    • Classify logging as to whether it’s appropriate to include in a customer build or not; for example, too much logging might bloat the size of your app unnecessarily, or slow things down even though the logging is disabled at runtime.

    • Construct your logging such that customer builds only get the logging you intend.

    • Provide a ‘secret’ way to enable that logging on the device; make sure App Review knows about this.

    • Provide a way for the user to send that log to you.

    • Throughout this process carefully consider both the security and privacy implications of your logging; for example, don’t log passwords and, if the user sends you a log, make sure the user understands what’s in that log.

Persistent Issues

If you still require help troubleshooting a Release build issue after reading this guide and exhausting its recommendations, use the following steps to contact Apple Developer Technical Support (DTS).

Related material

The following documents are linked within this guide and are listed here for your reference.

QA1747 - Debugging Deployed iOS Apps

QA1814 - Setting up Xcode to automatically manage your provisioning profiles

TN2151 - Understanding and Analyzing iOS Application Crash Reports

QA1693 - Synchronous Networking On The Main Thread

TN2285 - Testing iOS App Updates



Document Revision History


DateNotes
2016-09-28

Add app review and DTS contacts.

2016-08-05

New document that covers a breadth of error points that are commonly overlooked during app testing.