I've been running a test for 7 days now and the analytics keep saying that the conversion rate of the variant is identical to the baseline. I am just finding it very hard to believe that two variants can have an identical conversion rate every single day. I would expect there to be at least some variance.
Is this normal behavior? Has anyone else experienced this? Perhaps I need to wait longer?
Did you get an update on this? My baseline and update are showing as identical on Day One – which seems pretty unlikely when the difference is calculated with fraction of a percent precision.
Looking at your results, I'd say it's even more unlikely as that would require identical fluctuations each day.
What seems strange is that the table only shows a line for the test treatment, not the baseline treatment. If I were to guess, the dashboard is calculating the difference based on that same line item which, of course, will always show a zero difference.
I tried reaching out to support but they told me nothing was wrong. I tried making the same argument. There seems to be more variation when I did 2 treatments instead of one. And that makes even less sense since there's data per treatment when the volume of impressions has to be divided by more treatments. Still, perhaps you could try that.
I'm at a point where I really can't trust anything these metrics say. Hopefully more people will upvote this. I'm glad I'm not the only one. When I was talking to support I could just imagine them thinking I was crazy lol.
Wow – that must have been incredibly frustrating. I've contacted support, also. This feature should be disabled in its current state. I'm also getting reports that existing users who update – not just new users – are receiving the Icon treatment under test.
With your results, It's crystal clear to see that you're having daily fluctuations on the graph you share. The chance that the Treatment and Baseline would have the exact same fluctuations day-to-day is next to impossible. Clearly, either the value displayed for either the Baseline or Treatment under test is wrong and they're comparing either the baseline-to-baseline, or treatment-to-treatment. I'm curious to hear Apple's justification that this 'works as expected'.
I've noticed that if you tap the 'more' button and 'download table csv' you get both sets of data, but – for my app at least – while it now shows the number of impressions of each, the conversion rates are all estimates, and they're exactly the same for both treatments.
I wish it could include data such as taps/page views and downloads (similar to SearchAds). And the ability to filter results by source. I agree that this should disabled. It feels like it’s still in beta.
