Multivariate A/B testing and using child prebid configurations

This article describes how to use child prebid configurations as a way to perform multivariate A/B testing on a site.

A/B testing on a site is pretty straight-forward after understanding the concept of child prebid configurations. Using this concept, users can be split into two or more test-groups where you control how many % of visitors each group should contain. The groups can then be compared against each other using the headerbid analytics in order to determine the outcome.

Examples of parameters that can be used for A/B testing:

• Enabling/Disabling of one or more bidders.
• Changes of floor prices.
• Changes of headerbid timeouts.
• Switching between server-/client-side bidding.
• Custom execution of JavaScript code - to e.g. programmatically change Prebid.JS settings, load some other JavaScript, or whatever else.
• User-defined settings in Prebid Config Tag Fields.
• ..etc

Child configuration

A child prebid configuration creates as a "child" to a normal prebid configuration. Which SSP-placements that are selected is still decided in the normal ("parent") prebid configuration. This means that whenever the set of SSP placements for the normal configuration is changed - the child configurations will be updated automatically.

All Prebid parameters and Configuration Data settings in the parent configuration will be applied as well - but they are all possible to override in the child prebid configuration. To read about how settings are applied and in which order - read about Prebid parameters.

Introduction by example - benchmark enabling a bidder server- or client-side

In this hypothetical example we imagine that we've just got ourselves an account in Xandr  and want to test the performance (as measured by Ad unit eCPM). This is a good example where A/B testing can be beneficial as it's not always clear how much of added revenue that is "cannibalization" on other SSPs, and how much revenue that is lost because of the possibly longer load times when adding yet another SSP.

Preparation - Set the SSP as disabled as default before adding SSP placements.

This can for example be done globally. If necessary - read the introduction on Prebid parameters.

Step 1 - create child prebid configurations for the site we want to run the test

Select "Prebid configurations" on the Website/Apps level, and this will bring you the currently existing Configurations.

Select one of the existing configurations, which will take you to the option of creating child configurations:

In this test we'll create two child configurations, one called "Xandr client-side" and another called "Xandr server-side":

In this example we'll give them a Percentage of 25% each (although you might normally use a much smaller fraction of the traffic like 1%). Make sure to switch on Enabled child configuration when you want to start the test.

We're then ready to change the settings we want to change. As we don't want to only test individual placement(s) but all placement in the prebid configuration - select Prebid parameters on the Account/Publisher level (red marking above).

Note! If you would like to change settings for just one or a few placements, then open up the drop down, which lists all your placements, and then select the Placement you would like to do the change on, by pressing Prebid parameters.

Step 2 - change settings for the child configurations

Change the settings on the Account/Publisher level in the configurations (see above), so that we un-check the Disabled Bidder setting in the "Xandr client-side" configuration:

(If you want to do changes to other settings, like Floor prices or reloads, these can be found under the "Generic placement Data)

In the "Xandr server-side" configuration we'll do the same, but we'll also change Bidding type to Server-side:

Step 3 - Save settings and run the test.

That's it! You can now save the the settings and then save the Account/Publisher. Immediately after saving, the child configurations will be used - in this case for 25% + 25% of the visitors.

Step 4 - See the result of the test.

The Prebid configuration dimension in the HB Analytics header bidder reports can now be used to measure the performance of the test. The most important metric is normally the Ad unit eCPM metric as it shows how much revenue that is generated in average per ad request. You cna add your results as a Dashboard reporting module if you would like to keep a close eye on your test all the time, and even build alarms on it.

But there is also an A/B Test statistics section in the parent prebid configuration that shows  you the base metric Ad Unit eCPM, if you prefer not using the reporting or dashboards.:

We recommend that you utilize the reports and the Dashboard reporting modules, if you would like follow your progress over longer time periods though, but it is of course your own choice. In the reporting and dashboard reporting modules you can also filter for the configurations you would like to see:

Step 5 - Apply the "winning" settings.

In our example it turned out that the "Xandr server-side" configuration came out on top. What we probably would do in this case is to 1) disable the the Disable Bidder setting in the global settings (see the Preparation.. section earlier), 2) set Bidding type to Server-side and 3) delete the no longer needed child configurations.

However, in case the testing we're doing is only about optimizing the site/configuration we're currently testing - then we can copy the child configuration's settings into the parent configuration using the button outlined below:

This concludes the step by step guide, but if you would like to add more customized code to your child configurations and your tests, we recommend that you read through the section below.

Executing custom JavaScript code for a child configuration

Using Custom Config JavaScript Code we can execute any JS code.

This code will be executed once immediately when doing the first call to relevantDigital.loadPrebid().

NOTICE: As such code is only executed once per page-view / page-session it's worth to notice that in the case of a Single Page Application style page with different prebid configurations for e.g. frontpage/articles - then only the code for the first used configuration will be executed. If needed it's possible to create an own code-field with different behavior using Prebid Config Tag Fields.

Data available in the Custom Config JavaScript code

The following variables will be available in the JavaScript code in the field:

• data - An object with the Configuration Data for the prebid configuration. This includes the Built in settings (like the code itself). The keys for these built-in fields will be prefixed with "rlv". It also contain custom user-created settings created using Prebid Config Tag Fields.
• pbConfig - The prebid configuration object itself. It will be one of the objects returned by relevantDigital.getConfigs().