Framework library
A/B tests and randomised evaluations
How to use it
Specify the decision, outcome, assignment unit, comparison group and analysis window before launch. Compare all assigned units and report uncertainty.
Worked example
Assign checkout layouts before exposure and compare refund-adjusted contribution per assigned visitor.
Common pitfall
Conditioning on a post-assignment outcome can bias the result. Check spillovers and measurement gaps.
- Frame the questionWhat else could explain the observed difference?
- Apply the frameworkUse A/B tests and randomised evaluations on the worked example above.
- Produce an answerDefine the population, comparison, observation window and the evidence needed to rule out the strongest rival explanation.
Apply it to a case
Read the original framework guidance
Read the companion story: Two checkout pages and one misleading winner