Design an experiment around the decision
Write the decision and outcome first. Assign comparable units to alternatives, choose a measurement window and analyse people in their assigned group. Track harms such as refunds alongside the primary outcome.
Worked example
A checkout test assigns 2,000 visitors to each layout. One produces 180 paid orders and the other 200. The estimated lift is 1 percentage point, from 9% to 10%. Check uncertainty and refund-adjusted contribution before rollout.
Change the assumption
The original page earns ₹150 contribution per paid order. The new one earns ₹130. Which earns more per visitor in the example?
Check the reasoning
The original page earns ₹13.50 per assigned visitor and the new page ₹13.00, before refunds and other changes. Higher checkout conversion can coincide with lower contribution.
Check your reasoning
Why not compare only customers who reached payment?
Reveal the explanation
Reaching payment may itself be affected by the layout. Filtering after assignment changes the comparison groups and can hide drop-off.
Try the same framework in a different case
Explore worked stories in the reading room · Practise the next analysis