Documentation · Experiment Design
Setting Success Criteria
Define clear success criteria before launching experiments to ensure objective decision-making.
The Four Essential Components
1. Minimum Detectable Effect (MDE)
The smallest improvement that would be meaningful enough to justify rolling out the change.
- Simple changes (copy, colors): MDE can be lower (2–5%)
- Complex changes (new features): MDE should be higher (10–20%)
2. Statistical Significance Level
| Level | When to Use |
|---|---|
| 95% (p < 0.05) | Standard for most experiments |
| 99% (p < 0.01) | High-risk changes (payments, security) |
| 90% (p < 0.10) | Very low-risk, easily reversible experiments |
3. Required Sample Size
Use a sample size calculator with: baseline conversion rate, MDE, significance level (95%), and power (80%).
💡 Tip: The AI Metric Validator provides sample size guidance automatically when you validate your metrics.
4. Experiment Duration
- Run for at least one full week to capture weekly patterns
- Maximum 4 weeks - if no significance by then, the effect is likely too small
Decision Framework
| Result | Confidence | Action |
|---|---|---|
| Positive, significant | High | Roll out |
| Positive, not significant | Medium | Extend or iterate |
| Neutral | Low | Learn and move on |
| Negative, significant | High | Do not roll out |
