The Math of Feature Flags
A/B testing is applied frequentist hypothesis testing. The goal is to detect whether a localized change in the product yields a statistically significant change in a specific conversion metric, rejecting the null hypothesis.
Statistical Significance vs. Practical Significance
A result can be statistically significant (we are 95% confident the change is real) but practically insignificant (the change is a 0.01% lift, which doesn't cover the engineering maintenance cost of the code).
| Term | Definition in PM Context |
|---|---|
| P-Value | Probability of seeing this result if the feature actually had NO impact. |
| Statistical Power | Probability of correctly detecting a real effect if one exists (usually aim for 80%). |
| Minimum Detectable Effect (MDE) | The smallest lift you care about detecting. Smaller MDE requires exponentially larger sample sizes. |