Technical Note TN2249

Testing your app on Beta OS releases

This document covers the process of testing your apps on beta operating systems, as well as what to do when encountering changes in your app's behavior in the process. APIs and programming languages can evolve from release-to-release, which creates a need to verify your software continues to work as expected. To perform this verification, Apple provides registered developers with early access to the revised operating system before it goes public. Changes in a given update of the operating system might effect your app, so taking the time to check can either give your app an advantage over the competition, or, create a bad user experience for your customers.

Definition of Terms
Summary of things to look for
How to do it?
Best Practices
Test Unique Customer Configurations
What to do if your app behaves differently in the beta
Document Revision History

Definition of Terms

This document refers to the following relevant terms:

Summary of things to look for

The following is a categorization of the things you might encounter when testing your app on a beta OS release. These are the primary points of interest to look for when verifying your app works correctly in the new OS.

How to do it?

This section covers the required packages that must be downloaded and installed to begin testing your app on the beta.

  1. The beta OS must be installed on your device (Mac, Apple TV, Apple Watch or iOS device). The beta OS, when offered, is available on the respective developer download page. For example, see iOS Downloads.

  2. You must build and run your app using a version of Xcode that includes the beta SDK matching the version number of the OS you wish to test. Beta Xcode releases, when they are offered, are available on the Xcode Downloads developer webpage.

Best Practices

These are best practices to follow in general when testing your app on the beta.

Test Unique Customer Configurations

It is also important to be aware of the unique software configurations that your users may have. As an example, you should focus your efforts on the following software configurations:

  1. Testing on the previous operating system.

  2. Development and testing on the latest operating system your customers are using.

  3. Development and testing on the current beta.

What to do if your app behaves differently in the beta

Differences in behavior can result from a framework bug in the beta, or, a framework change that exposes a bug in your app's code. The following process should be used to distinguish between the two:

  1. Isolate the issue into a minimal reproducible case. By removing code not necessary to reproduce the bug, the cause of the problem is often easier to find. If the issue looks like a framework bug, the sample code you created in the process can more easily be attached to a bug report which is critical to expediting its fix.

  2. Issues with your app's code can be discussed on the Developer Forums: https://forums.developer.apple.com/ but remember to file a bug report for anything you think could be a framework issue.

  3. For framework bugs:

    • Its just as important to include a runnable Xcode project that reproduces the problem as it is to clearly describe the issue using the Bug Reporter: http://developer.apple.com/bugreporter.

    • Use the Developer Forums to discuss potential workarounds for framework bugs in the event that a fix does not make the first public release: https://forums.developer.apple.com/.

  4. Bugs that reproduce on the older, non-beta OS should be handled differently. Follow this process to deal with bugs that are present in a GM OS version:



Document Revision History


DateNotes
2016-05-20

Editorial update.

2015-09-02

Updated to include specific information about iOS, OS X and watchOS seeds.

2012-03-09

New document that covers the process of testing apps on beta pre-release Apple operating systems to catch early framework bugs or adapt to changes.