2 min lesson
Correlation, causation, reverse causation
Work through this case: "You find that workspaces with more agent errors also have higher churn. Before claiming errors cause churn, what alternative explanations must you rule out?" Say what you would do and why.
Step 1 of 2
Correlation, causation, reverse causationname which one you're claiming
- Pattern
- Correlation only
- Example on reliability data
- Heavy users have more errors because they make more requests
- How to tell them apart
- Normalize by exposure; the rate, not the count
- Pattern
- Reverse causation
- Example on reliability data
- "Errors cause churn" vs. churning users were already disengaged
- How to tell them apart
- Check temporal order; does the cause precede the effect?
- Pattern
- Confounded
- Example on reliability data
- A model swap and a traffic spike landed the same day
- How to tell them apart
- Hold the confounder fixed (segment, control or instrument)
| Pattern | Example on reliability data | How to tell them apart |
|---|---|---|
| Correlation only | Heavy users have more errors because they make more requests | Normalize by exposure; the rate, not the count |
| Reverse causation | "Errors cause churn" vs. churning users were already disengaged | Check temporal order; does the cause precede the effect? |
| Confounded | A model swap and a traffic spike landed the same day | Hold the confounder fixed (segment, control or instrument) |
Most bad causal claims are one of these three wearing a confident voice.
We see the error rate fell after the rollout, but two other things changed that week, so I can't call it causal yet. A diff-in-diff against the un-migrated region agrees with the raw drop and a back-of-envelope on the retry change predicts the same magnitude. Those three lining up makes me fairly confident and a clean switchback next week would settle it.
One method gives you one set of assumptions to attack. Three independent methods that agree force a skeptic to explain why all three would be wrong in the same direction, which is a much harder argument to make. Reaching for triangulation unprompted is a senior tell.
Don't defend a shaky claim to the death. The fastest way to fail the truth-seeking bar is to keep insisting on causality after the interviewer surfaces a confounder you can't rule out. Conceding "you're right, that confound means I can only claim association here and here's the experiment that would make it causal" scores higher than stubborn confidence.