Split testing calculator

Bayesian A/B experiments made easy  instructions

Your Data

Prior knowledge

Decision criterion

Control

Test

Control success rate: 
Test success rate: 

Approximate probability that test performs better than control:

Expected absolute change in success rate if test is chosen:

Recommendation:

* Note: You can always decrease the risk of making the wrong decision by collecting more data. If your sample size is small (less than a few hundred successes), or if it isn't representative of your population (e.g., it was collected over a short period of time), it's probably worth continuing the experiment. If your sample size is large and representative, but the difference between the control and test groups is negligible, it's probably worth moving on to other experiments.

** Recommendation explanation:

Control and test probability density functions

The success rate distributions for the control (blue) and test (red) groups. The distributions completely overlap if no data is entered, or if the counts for each group are identical. Success rates that fall within high density intervals are more likely than those that fall in areas of low density.

Difference between test and control

Distribution of differences in success probability between test and control groups. Obtained by simulating draws from the test and control distributions, where each sample is a possible success probability for the given group. Each control sample is paired with a test sample, and a difference sample is obtained by subtracting the control value from the test value.

Distribution intervals

The range of values contained in each central interval. For example, the first row shows the minimum and maximum values of the control, test, and difference distributions, for the 99% interval (i.e., where 99% of the values of each distribution fall – between the 0.5% and 99.5% percentiles). Note that the bounds for the difference distribution aren't necessarily the same as test minus the control bounds.

Why another calculator?

There are many split testing calculators out there. Those based on frequentist statistics, like Evan Miller's, assume a closed formula that requires setting the sample size in advance. Bayesian calculators, like Lyst's (which formed the basis of this calculator), let users encode their prior knowledge about the data, and do not require committing to a sample size in advance. This calculator aims to make Bayesian A/B testing more accesible by reducing the use of jargon and making clearer recommendations.

Questions/comments? Check out this post or drop me a line

Instructions

  1. 1Specify your prior knowledge. Set the success rate to what you think the average success rate is, and quantify your uncertainty (from 0 to 100, exclusive). For example, if you're running an experiment optimising the conversion rate of a landing page, and historical conversion rates are around 30%, set the success rate to 30, and use a low uncertainty value (e.g., 5).
  2. 2Specify the minimum effect that you care about. For example, if you only care about absolute changes in conversion rate that exceed 2.5%, set the minimum to 2.5.
  3. 3Enter the number of trials and successes in the control and test groups.
  4. 4Plot the distributions and choose whether to follow the recommendation. Note that each time you press calculate, you will get slightly different results due to the random simulation procedure. This is completely normal – it is very important to remember that few things are completely certain when it comes to experimentation.
  5. 5 Share your results by copying this page's link – it automatically changes to save your input when you press calculate.

Note: I tried to strike a balance between making this a useful tool for laypeople and providing rich information for the more statistically-inclined. Feel free to ignore greyed-out text like this if you don't want to dig too deep.