What Is an A/B Test?
An A/B test is a controlled experiment that splits traffic between two versions, a control (A) and a variant (B), to measure which performs better on a chosen metric. Randomly assigning users isolates the change as the cause of any difference, so the result can be trusted.
An A/B test is the workhorse of experimentation. It is also widely misused: most of the value is lost when teams stop too early or test changes too small to matter.
How an A/B test works
You take one population, randomly split it into two groups, and show each group a different version. Group A sees the control, the current experience. Group B sees the variant, the single change you are testing. Because assignment is random, the only systematic difference between the groups is the change itself, so any reliable difference in the metric can be attributed to that change. The discipline is in changing one thing at a time, otherwise you cannot say what caused the result.
How to run a valid A/B test
1. Start from a hypothesis
State what you are changing, what metric you expect to move, and why. A test without a hypothesis produces a number you cannot interpret.
2. Pick one primary metric
Decide the single metric that defines success before you start. Watching ten metrics invites you to cherry-pick a winner after the fact.
3. Size the test before launch
Estimate the sample size needed to detect the smallest effect worth acting on. Running until you see a result you like is how false positives happen.
4. Let it run to completion
Stop at the predetermined sample or duration, not the moment significance flickers. Peeking and stopping early inflates false wins.
Statistical significance and sample size
Statistical significance answers one question: how likely is it that the difference you saw is real rather than noise? The conventional bar is a p-value below 0.05, meaning under a 5 percent chance the result is a fluke. Significance depends on sample size and the size of the effect. Small effects need large samples to detect. Before you launch, decide the minimum detectable effect that would justify shipping the change, then size the test to find an effect that big. Skipping this step is the single most common reason A/B tests mislead teams.
A/B testing vs experimentation
An A/B test is a tactic. Experimentation is the system around it: forming hypotheses, prioritizing which to run, capturing what you learn, and compounding those learnings. A team can run hundreds of A/B tests and still not improve if nothing connects them. The test answers one question; the system decides which questions are worth asking and remembers the answers.
Frequently asked questions
How long should an A/B test run?
Run until you reach the sample size you calculated before launch, and cover at least one full business cycle, typically one to two weeks, so weekday and weekend behavior are both represented. Do not stop the moment significance appears, because early peeking inflates false positives.
What sample size do I need for an A/B test?
It depends on your baseline conversion rate and the smallest effect worth detecting. Smaller effects require larger samples. Decide the minimum detectable effect that would justify shipping the change, then use a sample size calculator to find the number of users needed per variant.
What does statistical significance mean in an A/B test?
It estimates how likely the observed difference is real rather than random noise. The common threshold is a p-value below 0.05, meaning under a 5 percent chance the result is a fluke. Significance alone is not enough; the effect also has to be large enough to matter.
What is the difference between an A/B test and a multivariate test?
An A/B test compares one variant against the control, changing a single element. A multivariate test varies several elements at once to measure their combinations, which requires much more traffic to reach reliable results. Start with A/B tests unless you have the volume to support multivariate.
Related terms
Go deeper
- A/B Test Management Tool
- Growth Experimentation Tools: Top 10 for 2026
- Growth Experimentation: The 2026 Ultimate Guide
About GrowthLab
GrowthLab is an experiment management tool where AI drafts the hypotheses, ICE and ROTI prioritize them, and every learning compounds into the next batch.
