Get started with product page optimization

Seema Vora: Welcome all and thank you for joining today's session on product page optimization.

My name is Seema and I'm on the App Store Monetization team based out of Cupertino. Cailin Arena: And I'm Cailin on the App Store Operations team. Seema: Today, I'll begin the session with a brief overview of the product page optimization feature on the App Store. Then Cailin will walk you through how to set up a product page optimization test in App Store Connect. After which, I'll share testing strategies to help you make the most of this feature. With that, let's get started. What exactly is product page optimization on the App Store? Product page optimization is a new feature that gives you the ability to optimize your on-store assets across the App Store, giving you a new way to improve your App Store conversion funnel. Each week, over 600 million visitors come to the App Store. Take a moment to let that number sink in.

You have a tremendous opportunity to showcase your app or game to many of them and drive downloads. We designed the product page optimization feature to help you maximize this opportunity. People visiting the App Store browse the Apps tab Games tab, and Today tab; search for specific apps on the Search tab; or come in through external referral links, such as social media posts or paid advertising.

Now with product page optimization, you can test different versions of your product page assets to learn which ones are most effective at getting App Store visitors to download or redownload your app or game.

In summary, here's an overview of the feature. You can test your app icon, different screenshots and app preview videos. You can choose the percent of traffic that will see the treatments for up to three treatments. You can also select specific localizations in which to run your test. You'll be able to submit your product page test assets for review, independent from the app binary, except for app icons, which must be included in the binary of the version being tested.

To help illustrate these important aspects of this feature, I'll use an example app, Mountain Climber. Here you see Mountain Climber's original product page. With the product page optimization tool, the elements that can be tested are: the app icon, as you see here, app preview videos and screenshots. Any other elements on the page will appear as they are on both the original and treatment pages and are not part of the test. If you have a banner on the top of your page, or you have in-app events featured on your page, they will appear on both the original and the treatment pages. Your product page optimization treatments appear across the App Store to people on iOS 15 and iPadOS 15 and later. To ensure people have a consistent experience across the App Store, your icon treatments -- meaning icons that you're testing -- will appear when people browse the App Store, like you see here on the left on the Today tab. Your icon treatment will also appear in Search, and of course, on your product page. Now, if you're testing screenshots or app preview videos, those treatments will appear in Search and on your product page. People will continue to see the same treatment across the App Store throughout the duration of your test.

Now, in line with that consistent experience, if you're running a test with an alternate app icon and someone downloads your app or your game from this treatment, The app icon used in that treatment will appear on their device. As with any optimization feature, you need analytics to help you understand which treatment performs the best. We've expanded App Analytics to have a new product page optimization dashboard that'll provide you with the data to help you identify which treatment performs the best. As part of this feature, we've developed a unique methodology that enables you to continuously monitor your test while it's running, meaning you can identify exactly when it's time to make a decision and stop the test at any time. The user interface includes indicators to help guide your analysis as your test progresses. These indicators tell you when a treatment is likely performing better or worse than your original product page, or whether more data is needed before you should make a decision. We'll cover more detail on App Analytics later in the presentation.

So how does this product page optimization feature benefit you as a developer? As a start, we've streamlined the submission process with a new submission flow specifically designed for this feature, which allows you to submit your product page test assets for review, independent of the app binary in App Store Connect, except for app icons, which must be included in the binary of the version being tested. The product page optimization tool makes applying the better-performing creative as easy as possible, so you can apply better-performing screenshots or app preview videos within App Store Connect. You can access updated tools in App Analytics to monitor metrics like unique impressions, conversion rate, and performance results of the test to help you make data-driven decisions. As you can see, this feature benefits you as a developer and gives you the unique opportunity to test within the App Store's native environment. We've created a robust set of tools to help you set up product page optimization testing. Now I'll hand it over to Cailin who'll show you how to implement this feature in App Store Connect. Cailin: Thanks Seema. Before we begin the test setup walkthrough, make sure that your app is ready for sale, and that you have the account holder, admin, marketing, or app manager role.

Start by clicking on Product Page Optimization on the left navigation, under the Features subsection. Then click Create Test to begin setting up your test. Here, you'll start by entering in some basic information, starting with reference name, which is how you'll identify this test going forward.

In this example, we'll give it the name "Preview Video A/B Test." Next, you'll choose the number of treatments. You can have up to three treatments. For this example, we'll go with one. For Traffic Proportion, this is the percentage of users that will be randomly shown a test treatment instead of your default product page. For example, if a test has three treatments and you choose a 30 percent traffic proportion, each treatment will be shown to 10 percent of the total traffic. In this example, since we chose one treatment, we can pick anything from one to 50 percent. We'll choose 50 percent to allow the test to generate significant results in the shortest time possible. For localizations, you have the option to run this test on only a subset of your app's localizations. We'll keep all localizations selected by default, to allow for the maximum number of users to participate in the test. From here, you can optionally get an estimate of how long this test will take by expanding the Estimate Your Test Duration section. The estimate is based on your app's product page performance over the last week and a target threshold of 90 percent confidence. You just need to select an improvement value to get the estimate.

For example, if we choose five percent, this will calculate the number of impressions needed and an estimated duration to get those impressions, based on the last week's performance. In this case, it would take about 174,404 impressions -- or about two weeks' time -- to conclude that the treatment results in 5 percent improvement over the original product page, with 90 percent confidence. After reviewing this, click on the Create Test button. Now, you'll be able to set up your treatments. By default, the treatment will be titled "Treatment A," but you can edit that name to anything you like. The treatment will start off as an exact copy of the original page -- same icon, screenshots, and preview videos; although in this case, our Mountain Climber app currently does not have an app preview. So we could add an app preview to Treatment A for each applicable device size, and then click Start Test.

Then click Add for Review.

Here, you'll see a submission confirmation screen, showing the app version that will be used for review, and the items that will be reviewed. You could choose to add other items to this submission, but for this example, we'll click to submit just this test. And with that, your test will be submitted to App Review. All metadata being used in a test must be accepted by App Review before use. If you try to start a test before the metadata is accepted, the test won't start and the metadata will be added to a submission for you to send for review. If you already have a submission for that platform in review, you'll need to wait until the first submission is complete before you can send the next test for review. Going back to product page optimization, you'd see your test with the status "In Review." Once App Review does approve the new assets, the status will automatically update to "Running," and show how long the test has been running for. At this point, you could click View Analytics to jump over to App Analytics. Here, you can see how the test is going. You'll see the conversion rate for the original page and the treatment page at the top, and a conversion rate trend chart below. Once the test has reached enough impressions to reach 90 percent confidence, the dashboard will indicate how each treatment is performing. In this example, enough data has been collected to suggest that Treatment A is performing better, with a 1.9 improvement over the original page's conversion rate. At this point, you may want to apply Treatment A to your original product page. To do this, you'd go back to the test by clicking View in My Apps in the bottom left.

Here, you can click on Apply Treatment to Original Product Page to add the app preview video. You can choose whether to apply the treatment to the version live on the App Store, or another version in Prepare for Submission, if you have one. Then click Apply Treatment. As a result, the app preview we had in treatment is now live in the App Store with version 3.1, and it's also been applied to version 3.2 here. Now, if you wanted to run a test on the app icon, you'd need to create a test on that app that includes multiple icons in its binary. If your app has multiple icons included in the binary, your treatment setup page will include an App Icon tab. And then click Change to select a different icon. All app icons included in the app binary for the version of your app currently on the App Store will display in the dropdown. To include them, you must be using Xcode 13 or later, which includes the latest SDK that supports alternate icons in asset catalogues. From here, you could click Start Test and you'd be able to start the test without going through App Review, because the icon has already been reviewed when it was included in the binary for the currently live version. When it's time to promote a treatment with an app icon to your product page, you'll need to create and submit a new app version. Finally, the App Store Connect API will be valuable if you want to integrate product page optimization with your own systems. The API will allow you to automate the creation and management of tests outside the App Store Connect website. I'll now hand it over to Seema to share how you can pull all this together with product page testing strategies. Seema: Thank you, Cailin. As Cailin mentioned, I would like to go over testing strategies. Basically, what you should be thinking about as you plan your product page optimization testing and experimentation.

As with any marketing initiative, we'll start with looking at the top of the funnel. Here you see the traffic mix for the App Store Funnel. Important to keep this traffic mix in mind as you think through your testing strategies. As you create your testing plan, it's important to note that your optimization efforts are not a one-and-done. It is an iterative, ongoing process. As you create your test, start with thinking of which asset matters the most. For some, it might be figuring out the best possible icon; for others, it might be the addition of an app preview video. As you configure your test, ensure that the testing variations are notable and they're significant enough to make a difference. Minor tweaks tend not to yield conclusive results. Also, not all experiments will produce positive results. If so, reevaluate the testing plan and test again. Be sure to analyze your test's performance until you've identified the treatment that performs best. Once conclusive, use the results to inform next steps and update your product page with the better-performing creative. The product page optimization tool makes this easy by letting you update the screenshots and app preview videos after the completion of a test right there within App Store Connect. Now, I'd like to share some ideas to help jumpstart your product page optimization test planning, starting with examples to help you think through testing asset testing. First off, we'll go over testing by feature. For example, does testing a screenshot of an existing feature drive better conversion than a screenshot showcasing a new feature? Or say you are a fitness app. Does highlighting your primary value proposition help increase your conversion rate? You might test by theme. We've heard that many of you want to test seasonal content. Now, you can test to see if updating your icon or assets to reflect seasonality improves your download conversion rate or not. You could also test by character or new content. For example, an entertainment app could test to see if showcasing new content performs better than classic content. And lastly, you might test different creatives. This one's a bit nuanced but can be very effective. For example, does adding an app preview video help improve your download conversion rate? Or testing color variations like varying the background color of a screenshot to see which background color performs best. Keep these examples in mind as you plan your product page treatments. Now that we've discussed some asset testing ideas, here are a few important considerations to help you create a test plan starting with traffic proportion, which is the percentage of people who will randomly see a treatment version of your product page assets. Think about how you want to distribute this percentage between your original product page and treatments. When allocating traffic to treatments, keep in mind your conversion rate. You can choose to allocate more traffic to your original product page, which is your control, in case you're concerned about how a new treatment might perform. Do note that no single treatment can receive more traffic than your original product page. Another aspect to consider is planning your assets and treatments. Think through the product page asset you will include in your test and number of treatments you'll create. We do recommend testing one asset at a time to ensure that the results are not cluttered. As a reminder, you can create up to three treatments per test. Localizations. The product page optimization feature includes testing for specific localizations. Think through what localizations might be worth including in your test...

such as those locales you're interested in expanding your app or your game's market reach. Although important to keep in mind the traffic volume from those locales, as that could result in your test taking a bit longer to get meaningful results. Finally, think through the duration of a test. When setting up your test in App Store Connect, the tool will give you an estimate based on your app's product page performance over the last week and a target threshold of 90 percent confidence. You just need to select an improvement value to get the estimate. So keep these considerations in mind as you plan your testing to ensure maximum efficacy of your product page optimization efforts.

Let's take a look at a few sample tests. As a developer, here is my sample test plan. This sample test plan includes the title of the test, test description, hypothesis, number of treatments, expected traffic distribution, and what localizations the test should be applied to. The first sample test is an icon color test where we want to test three color variations for the app icon versus the control. As icons are universal, we plan to test this across all locales. Our hypothesis here is that the brighter icon will perform better. The second sample test is testing localized app preview videos for the Japanese market, and our hypothesis here is that locally relevant assets will drive higher conversion. Since this is a market-specific test, the only localization we'll select here is Japanese. The third sample test is testing out different screenshots to see if character-focused visuals perform better. And in this case, we'll allocate 40 percent of our product page traffic to our treatments, leaving 60 percent to see our control, which is our original product page. Let's take a closer look at that first test, which is an icon color test where we will test icon color variations across three treatments, with a traffic distribution of 30 percent to the treatments across all localizations. Let's say we're a game developer and we want to test out different icons for our game The Coast. Here's what our original product page looks like. On the left, you see our original treatment -- the one we just looked at -- and on the right, you see the three treatments we've created for our test, each with a different color icon. As you saw in the test plan, we chose to show our treatments to 30 percent of App Store traffic, which means that each of our three treatments would receive 10 percent of the total traffic; and the original will receive 70 percent of the total traffic. Once our test metadata is approved by App Review, the test will start. Let's take a look at what this looks like in App Analytics as our test is running, starting with the results dashboard. At the top, you will see your test name, the date your test began, and the current status of your test. Below that, you'll see an overview of your test performance, with the conversion rate for your original product page as well as for each treatment included in your test. The Conversion Rate Trend chart visualizes how your test is progressing over time. You can see the natural ebbs and flows as your test accumulates data and begins to determine which treatment performs best. And at the bottom, you'll get even more details about your test, including the number of unique impressions each treatment has received, the conversion rate for each treatment, the improvement each treatment has achieved compared to the original product page -- also known as the baseline -- and the confidence level that we have in the results. During the first seven days, we will not show a confidence level in order to account for any daily fluctuations before providing a result. After your test has been running for at least seven days, you'll see confidence indicators appear. Now in this test, we have 72 percent confidence that Treatment A is outperforming the baseline, 58 percent confidence that Treatment B is outperforming the baseline, and very low confidence that Treatment C is outperforming the baseline. Until a treatment reaches 90 percent confidence in the result, you'll see "Collecting Data" for these treatments. Now, if a treatment reaches 90 percent confidence, relevant indicators will appear under each treatment. For example, at this point in the test, Treatment B is performing worse than the baseline with at least 90 percent confidence, while Treatment A is performing better. And Treatment C is performing so similar to the baseline, that it's unlikely we'll be able to detect a significant difference within the 90-day test duration if the results continue to accumulate at the same pace. If you see this status for all your treatments, we recommend you end the test and try again. Now since the test status indicators are a new addition to App Analytics, let's go over each one to understand what they mean. The original product page is the baseline by default, which is the treatment that all others are compared against, Collecting data means that we're not yet able to determine whether the treatment is performing better or worse than the baseline with 90 percent confidence or more. "Performing Better" means that the treatment is performing better than the baseline with at least 90 percent confidence. "Performing Worse" means the treatment is performing worse than the baseline with at least 90 percent confidence. And "Likely to Be Inconclusive" means that the test isn't getting enough data to make a determination within the 90-day limit.

If you want to compare treatments to one another, you can change the baseline. For example, here we can compare each of our variants against Treatment A. In this case, each variant is performing worse than Treatment A with at least 90 percent confidence. This means that you can be confident that Treatment A is indeed the best-performing treatment out of them all. Now going back to our icon test with the original product page as the baseline. By clicking on the test name, you can easily swap between your current test and past tests. You can also export the data in the chart or the table as a CSV so you can analyze the data on your own. If you want even more details about your results, you can hover over the conversion rate to view the credible interval, which shows the range of possible values that we estimate for each treatment. This information helps you get a deeper understanding of the results.

Lastly, once you're ready to promote the better-performing treatment to your default product page, you can click on the link at the bottom to navigate back to My Apps. This wraps up our sample icon test, including interpreting and evaluating the results in App Analytics. I hope that you now have a comprehensive understanding of the product page optimization feature, enough so you can start planning your own testing. Before I conclude, I'd like to go over a few things to look out for as you implement your testing plans. Not reaching statistical significance. It is important to remember not to apply results based on early testing or those with minor uplifts. As a reminder, App Store Connect will let you know how long the test will take based on your traffic distribution and your current conversion rate. It can be tempting to act on early results. It's important to not make any conclusions until your test has reached statistical significance. This ensures that you can confidently claim that the winning result will make a notable and positive difference to your conversion rate. Make sure the testing treatment has significant enough variations from the control, if not, the test could result in being inconclusive. And ensure that the test is set up to succeed. Cross-check that culturally relevant or regional assets are applied to the right localizations, and review the estimated duration of the test to ensure that the traffic from these localizations are sufficient enough to drive results within the 90-day testing timeframe. As you execute your testing plan, keep these in mind to help avoid any delays or any issues with your product page optimization testing.

If you're looking for more information or would like to reference what we discussed today, do visit our Developer website, which is an excellent resource that you can reference at any time. The Developer website has helpful information about the feature and considerations for when setting up your tests. Once you start testing, we recommend you continue to identify new conversion goals and areas where your conversion rate can be improved. With that, I'd like to conclude our product page optimization session today. We are excited to see how you all will use this feature to improve your App Store conversion rate over time. Thank you.