Split testing calculator

Bayesian A/B experiments made easy  instructions

Control and test probability density functions

The success rate distributions for the control (blue) and test (red) groups. The distributions completely overlap if no data is entered, or if the counts for each group are identical. Success rates that fall within high density intervals are more likely than those that fall in areas of low density.

Difference between test and control

Distribution of differences in success probability between test and control groups. Obtained by simulating draws from the test and control distributions, where each sample is a possible success probability for the given group. Each control sample is paired with a test sample, and a difference sample is obtained by subtracting the control value from the test value.

Distribution intervals

The range of values contained in each central interval. For example, the first row shows the minimum and maximum values of the control, test, and difference distributions, for the 99% interval (i.e., where 99% of the values of each distribution fall – between the 0.5% and 99.5% percentiles). Note that the bounds for the difference distribution aren't necessarily the same as test minus the control bounds.

Instructions

  1. 1Specify your prior knowledge. Set the success rate to what you think the average success rate is, and quantify your uncertainty (from 0 to 100, exclusive). For example, if you're running an experiment optimising the conversion rate of a landing page, and historical conversion rates are around 30%, set the success rate to 30, and use a low uncertainty value (e.g., 5).
  2. 2Specify the minimum effect that you care about. For example, if you only care about absolute changes in conversion rate that exceed 2.5%, set the minimum to 2.5.
  3. 3Enter the number of trials and successes in the control and test groups.
  4. 4Plot the distributions and choose whether to follow the recommendation. Note that each time you press calculate, you will get slightly different results due to the random simulation procedure. This is completely normal – it is very important to remember that few things are completely certain when it comes to experimentation.
  5. 5 Share your results by copying this page's link – it automatically changes to save your input when you press calculate.

Note: I tried to strike a balance between making this a useful tool for laypeople and providing rich information for the more statistically-inclined. Feel free to ignore greyed-out text like this if you don't want to dig too deep.