A/B Test Significance Calculator

What the confidence number means

This calculator runs a two-tailed two-proportion z-test, the standard check for comparing two conversion rates. Confidence is the probability that a difference this large would not have appeared by chance alone if the two variants were genuinely identical. At 95% confidence there is roughly a one-in-twenty chance the result is noise, which is the threshold most teams treat as the minimum before shipping.

Two-tailed matters. It tests whether B differs from A in either direction, rather than assuming in advance that B is the winner. A one-tailed test reaches significance sooner, which is tempting and is exactly why it is so often misused.

What significance does not tell you

Confidence answers whether a difference is real, not whether it is worth having. A statistically significant 0.3% lift on a low-value page can be real and still not worth the engineering to ship it, while a large lift at 80% confidence may be worth running longer rather than discarding. Significance is also not a stopping rule: checking daily and shipping the moment the number crosses 95% inflates false positives badly, because you have effectively run many tests. Decide the sample size and duration before you start, run through at least one full business cycle so weekday and weekend behaviour are both represented, and read the result once.