Two checkout pages and one misleading winner
In a fictional checkout test, the team sees fewer refunds on a new page. The sales lead wants to roll it out. The analyst asks for the number of visitors who saw each page. The new page may have fewer refunds because fewer visitors complete an order.
What the analyst checked
Use a denominator tied to the decision. Refund rate among orders describes order quality, while net contribution per assigned visitor captures the commercial effect of both checkout and refunds. Analyse each visitor according to their assigned page, including those who never reached payment.
The next decision
Compare both metrics over the same observation window. Report uncertainty and any guardrail breaches. Decide whether the effect is large enough to matter after the extra costs of serving orders and handling refunds.