hardScenarioCase StudyMidSeniorStaff
Your headline metric went up and the business got worse. How does that happen, and how do you catch it?
4 min readUpdated 2026-08-15
Answer
Short answer
Ratios improve when the denominator shrinks and averages improve when the mix shifts, so a rate can rise while every underlying cohort worsens; you catch it by pairing every rate with its absolute counts, decomposing by segment, and defining guardrails before launch.
Was this answer helpful?
Likely follow-ups
- Which single metric would you add to a dashboard that only reports rates?
- How would you decide whether the improvement is worth the volume you lost?
- You find the mix shifted. How do you report that to a leadership team expecting good news?
- When is a shrinking denominator genuinely the outcome you wanted?
Related questions
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?hardAlso on guardrail-metrics5 min
- Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?hardAlso on metrics5 min
- Daily active users dropped 15% week over week. How do you diagnose it?mediumAlso on metrics4 min
- Leadership has asked for 20 per cent growth in activations next quarter. How do you decide whether that is the right target?hardAlso on metrics5 min