This free Bayesian A/B test calculator will help you compare two variants of your website, A and B, or two interventions, and tell you the probability that B is better. You can read more about A/B testing in our earlier blog post on the subject. You may also be interested in our Chi-Squared sample size calculator which will help you calculate the minimum sample size needed to run a Chi-Squared test, given an expected standardised effect size.

Assuming you have already run your A/B test, you will have recorded for both variants of your website, A and B, the following data:

1. How many people have visited that variant

2. How many people converted on that variant (this could be a click, a purchase, a contact request, or any other desirable outcome)

Simply enter the numbers that came out of your A/B test into the calculator below:

How does a Bayesian A/B test calculator work?

The calculator relies on Bayesian simulations. It simulates 100,000 samples from a distribution and uses this to calculate the approximate probability that B beats A. The conversion rates for both variants are treated as random variables following a Beta distribution.

Here are the steps of how the Bayesian A/B test calculator calculates the probability that version B is better than version A.

1. Start with a clean slate: the Bayesian A/B test calculator takes a prior assumption that version A and version B are equally good, before looking at any data. This is called a “flat prior”. By changing alpha and beta in the code sample below, you could start with a more informed prior, if you had a strong reason to believe that the two versions are different (perhaps from historical data).

2. Update with real results: The calculator then takes the actual results - the visitors and conversions for both Version A and Version B - to update its understanding.

3. Builds a “map” of possibilities: Using a mathematical formula (the Beta distribution), the calculator creates a range of likely conversion rates for each version based on the data provided.

4. Run a massive simulation: To see how they compare, the calculator “races” the two versions against each other 100,000 times by randomly picking possible conversion rates from those maps.

5. Calculate the winner: Finally, it counts how often Version B came out on top during those simulations to give you a percentage of how confident you can be that B is truly better than A.

Why is a Bayesian A/B test better than a Chi-Squared test or t-test?

I would not go so far as to say that Bayesian A/B test is universally “better” than a Chi-Squared test or t-test. However, Bayesian methods have become the favorite in modern tech because they answer the questions business owners actually ask. Chi-Squared and the t-test are methods from the classic frequentist school of statistics and there is a long-running debate between statisticians who prefer frequentist methods and those who prefer Bayesian methods.

There are a number of advantages for using Bayesian A/B tests in a business context in particular:

1. A Bayesian A/B test gives you a “probability of being better”, e.g. there is a 94% chance that Version B is better than Version A. This is easier to work with in a business context.

2. You can stop a Bayesian A/B test early (“peeking”). With Chi-Squared, you are technically supposed to decide your sample size before you start and only look at the results once. If you “peek” at the results every hour and stop as soon as it looks good, you significantly increase the risk of a “false positive.”

3. A Bayesian A/B test incorporates “prior knowledge”. If you know your baseline conversion rate is usually 5%, you can bake that into the maths so a small, random fluke in the first 50 visitors doesn’t skew your results.

4. A Bayesian A/B test allows you to quantify things like “even if A is actually better, switching to B will likely only cost us 0.01% in conversions.” This helps you decide if the risk of switching website designs, models, or interventions, is worth the potential reward.

While Bayesian methods are becoming commonplace in digital marketing, the chi-squared test and t-test remain the gold standard in scientific research. If you’re publishing in a peer-reviewed journal, you probably want to use a statistical test from the frequentist toolbox. Chi-squared, the t-test, and other frequentist tests output p-values, which are a useful standard which can be compared across research papers. However, if you need to explain the cost or expected return of switching a website design to a non-statistician business owner, Bayesian methods would be my recommended tool.

Python code

So that you can understand how the Bayesian A/B test probability is being calculated, the Python code will appear below when you run the Bayesian A/B test calculator.

First you need to install some dependencies:

pip install scipy numpy

# Python code will appear here.

R code

You can also run the code in R:

# R code will appear here.