2 min lesson
On-call without burning the team
Start at the first move in "On-call without burning the team" and carry it through to the proof.
Step 1 of 2
On-call without burning the teamdata on-call is different from service on-call
Data incidents are often slower-burning than a downed API. A late table at 2am rarely needs a human at 2am if the consuming dashboard is read at 8am. Encode that: severity tiers decide who gets paged and when.
- 1Severity-tier every alert. Sev-1 pages immediately (revenue/agent data broken). Sev-3 opens a ticket for business hours.
- 2Route by ownership. The alert goes to the team that owns the asset, not a shared firehose nobody reads.
- 3Dedupe and group. One upstream break should fire one incident, not forty downstream alerts that bury the root cause.
- 4Write the runbook before the incident. Each tier-1 asset has a runbook: how to detect, how to backfill, who to escalate to.
- 5Degrade gracefully. Serve last-good data with a staleness banner rather than a hard failure where you can.
An on-call who's been paged six times for a non-issue will miss the seventh page that matters. Every alert that fires and gets dismissed without action is a bug in your alerting, not a fact of life. Track alert→action rate as a metric and prune anything that's mostly noise.
"I'd tier datasets by what they feed - revenue and agent-training data get minute-level freshness SLAs and paging; exploratory tables get best-effort and a dashboard. Then I report platform reliability as a first-class number to stakeholders each month, so reliability is a shared metric, not a thing they only notice when it breaks."