1 min lesson
Slicing and the traps
Explain the practical point behind "An aggregate metric is where pain goes to hide."
Step 1 of 4
An aggregate metric is where pain goes to hide. The metric-design round will hand you a number that looks healthy and watch whether you trust it or interrogate it.
The reflex they want is segment before you conclude. A global success rate of 94% can mask a brand-new client version sitting at 60% or one model that is quietly failing on large repos. The aggregate is an average of experiences and averages drown out the people who are suffering.
Learn more
Advanced table
The four traps
The four trapsnamed, with the tell for each
- Trap
- Simpson's paradox
- How it bites
- Global metric improves while every segment worsens, due to mix shift
- Defense
- Always check segment-level trends, not just the aggregate
- Trap
- Survivorship / selection bias
- How it bites
- Users who churned from a bad run stop logging events, so the survivors look fine
- Defense
- Track cohorts from a fixed entry point; don't condition on still-active
- Trap
- Heavy tails
- How it bites
- Means are dominated by outliers and hide where pain lives
- Defense
- Report medians and percentiles; watch variance, not central tendency
- Trap
- Mistaking volume for impact
- How it bites
- Ranking issues by raw event count buries a small group in severe pain
- Defense
- Rank by affected-user blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition., not by event frequency
| Trap | How it bites | Defense |
|---|---|---|
| Simpson's paradox | Global metric improves while every segment worsens, due to mix shift | Always check segment-level trends, not just the aggregate |
| Survivorship / selection bias | Users who churned from a bad run stop logging events, so the survivors look fine | Track cohorts from a fixed entry point; don't condition on still-active |
| Heavy tails | Means are dominated by outliers and hide where pain lives | Report medians and percentiles; watch variance, not central tendency |
| Mistaking volume for impact | Ranking issues by raw event count buries a small group in severe pain | Rank by affected-user blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition., not by event frequency |
If asked "the global metric went up, ship it?", do not say yes. Say: "Up globally can hide every segment going down if the traffic mix shifted - I'd confirm each major segment moved the same direction before I conclude." That single sentence is what the round is fishing for.