A/B testing

An A/B test is an experiment in which the current settings apply to one part of the impressions in the Yandex Mediation unit (Control), and alternative settings to the other part (Test). Based on the test results, you can compare how effective Control and Test settings are and decide whether to keep the current settings for all impressions or apply new ones.

Creating an A/B test

To create an A/B test:

1. Open the settings of the unit you want to run the test for.

2. Go to Networks and click Create A/B test. Settings for Test will open.

3. Enter the name of the A/B test.

4. Choose the size of the test group (from 10% to 50%): this is the percentage of traffic that will go to Test.

5. Specify the test period. The recommended period is at least 7 days.

6. Configure alternative settings for Yandex Mediation. What you can change:

   • Redefine ad sources by adding or removing ad networks in the unit.
   • Change the Smart Waterfall sequence by experimenting with eCPM values for different ad networks.

   Note

   We recommend changing one setting for one test (for example, add one ad network and leave other settings unchanged): this makes it easier to analyze the results.

7. Click Save.

You can't change unit settings during the test.

Analyzing test results

To check the test progress and view the results, go to Ads in apps → A/B tests. Click  next to the test name. Data presented in two parts will open:

Summary table

Contains general information about the test:

• Profit: The difference between the remuneration from Test impressions and remuneration from Control impressions. By default, the difference is indicated in rubles, but you can change the currency to US dollars or euros. Currency is converted at the exchange rate of the Central Bank of the Russian Federation on the day of the impression.
• Impressions: The total number of impressions during the test. The more impressions, the more representative the test results. We recommend analyzing the results when the total number of impressions is more than 10,000.
• Days: Test duration from the start date to the present.
• Test group allocation: The percentage of traffic that goes to Test.

Comparison table

The table contains separate statistical indicators for Control and Test, as well as for ad networks within Groups:

• Impressions: The number of impressions of the Group or ad network during the test.
• eCPM: The cost of one thousand impressions of a Group or ad network.
• Fill rate: The ratio of the number of served ads to the number of requests (as a percentage) within a Group or ad network.
• Show rate: The ratio of the number of impressions to the number of served ads (as a percentage) within a Group or ad network.
• Revenue: The amount of remuneration accrued to the partner for Group or ad network impressions. It is indicated in the same currency as the profit in the summary table.

Depending on the results, complete the test:

• If Test generates more revenue, we recommend applying the new mediation settings to all impressions in the unit.
• If Test generates less revenue, we recommend returning the original mediation settings for all impressions in the unit.

Test statuses

Running tests have the Active status.

If the test period has ended, but no decision has been made based on its results, the test status changes to Expired.

Completed tests have the Stopped status.

Contact support