A/B Testing Sample Size Calculator

Calculate the number of trials needed to see a change of desired delta % from a baseline conversion %.

Baseline Conversion % (c):

Desired Delta % (d):

Number of Trials (N):

Calculate the delta % observable given a number of trials and a baseline conversion %.

Baseline Conversion % (c):

Number of Trials (N):

Observable Delta % (d):

A/B Sample Size Formula

There are two variations of the same formula here - rearranged in order to calculate the desired output: The number of trials required to observe a percentage change or to calculate the change (delta) that could be observed given the number of trials run.

Both at the 95% confidence level and 80% statistical power - which are industry standard levels.

$N = 16 \times \frac{c - c^{2}}{d^{2}}$

Or rearranged to give:

$d = 4 \times \sqrt{\frac{c - c^{2}}{N}}$

Examples

For a baseline conversion percentage of 4% and a desired delta percentage of 2% the number of trials needed is 1500.

$1500 = 16 \times \frac{0.04 - 0.04^{2}}{0.02^{2}}$

Whereas the change observable with 24000 trials and a base conversion of 2.8% would be 0.4%.

$0.004 = 4 \times \sqrt{\frac{0.028 - 0.028^{2}}{24000}}$

Background

Calculating the number of trials for an A/B test to achieve a statistically significant result is dependent on the size of the effect (the delta) you hope to observe - with smaller deltas requiring a larger number of trials to effectively observe the change.

This calculator page simiplifies the process by standardising the calculations and using a simplified formula.