Your headline metric went up and the business got worse. How does that happen, and how do you catch it?

Two mechanisms, and they are not the same

Almost every version of this story is one of two mechanisms, and naming which one you are looking at is most of the answer.

The first is denominator collapse. Any metric expressed as a rate has two ways to improve, and only one of them is good news. If signup-to-paid conversion rises from 4 per cent to 6 per cent, that is excellent if paid customers went up and arithmetic if signups went down. Take a month with 10,000 signups and 400 conversions, then a month with 5,000 signups and 300 conversions: conversion has risen by half while new customers have fallen by a quarter. Nobody lied, nobody misreported, and the dashboard is genuinely green.

The second is mix shift, where the aggregate moves in the opposite direction to every part of it. Suppose 30-day retention among self-serve users is 40 per cent and among sales-assisted users 80 per cent, and the split of new accounts is even, so blended retention is 60 per cent. Now a quarter later self-serve retention has fallen to 38 per cent and sales-assisted to 78 per cent, but the mix has shifted to 80 per cent sales-assisted because self-serve acquisition was cut. Blended retention is 0.8 times 78 plus 0.2 times 38, which is 70 per cent. Retention improved by ten points and got worse for every single user. This is Simpson's paradox, and using the name is worth a little credit, but constructing the example is worth much more, because it demonstrates you would spot it rather than recognise it.

What the diagnosis looks like

Work in a fixed order, because the order is what is being graded. First, get the absolute numerator and denominator for the metric in both periods. This single step resolves denominator collapse immediately and it is astonishing how often it is skipped in favour of hypotheses about user behaviour.

Second, decompose the metric along the dimensions that could have changed composition rather than behaviour: acquisition channel, plan tier, geography, platform, and account age. A rate that improves in aggregate and worsens within every segment is diagnostic of mix shift and nothing else. Hold the mix constant at the earlier period's weights and recompute — in the retention example, applying the original even split to the new segment rates gives 58 per cent, which is the honest comparison and shows a two-point decline.

Third, look at what changed upstream. Mix shifts do not happen spontaneously; something caused them, usually a change to acquisition spend, a pricing or packaging change, a change in eligibility for the flow being measured, or a filter added to the instrumentation. That last one matters: if the change under investigation also altered who enters the measured funnel, the before and after populations are not comparable at all and no segmentation will rescue the comparison.

The version that is not an artefact

There is a third case that candidates often miss because it looks like an error and is not. Sometimes the metric is measuring exactly what it claims and the trade-off is real. A checkout flow that adds friction to filter out fraudulent orders will cut conversion and improve margin. A recommendation change that increases sessions per user by surfacing outrage-adjacent content will also increase complaints and eventual churn. An onboarding change that gates a feature behind a payment step will raise revenue per activated user and reduce the activated population.

In those cases nothing is broken about the measurement, and the finding is that the metric was the wrong choice of proxy. The correct response is to escalate to whatever the metric was standing in for, usually retained revenue or retained users over a horizon long enough to contain the damage, and to say plainly that the team optimised a proxy that had come loose from the thing it proxied. This is the finding leadership least wants and most needs.

Preventing it rather than detecting it

The preventive discipline is cheap and rarely done. Every rate on a dashboard should be reported next to its numerator and denominator, so a collapsing denominator is visible without anyone thinking to look. Every experiment or launch should declare guardrails before it ships, and the guardrail must be an absolute count on the other side of the trade-off — total paid conversions, not conversion rate; total activated users, not activation rate. And any metric that is a blended average across segments with very different levels should be reported segmented as a matter of routine, because that is the only structural defence against mix shift.

The trap here is the reflex to explain rather than to verify. A metric moving in a pleasing direction is the case where nobody demands the denominator, and the whole reason this failure survives long enough to hurt is that it arrives disguised as success. Treat a surprisingly good number with the same suspicion you would give a surprisingly bad one, and say so in the interview, because that instinct is the actual signal being tested.

A rate has two ways to go up and only one of them is a win, so the first question about a good number is the same as the first question about a bad one — what happened to the denominator.