Skip to lesson
Exit
Deep Dive - Webhooks, Events & Reliability1 / 2

1 min lesson

Designing the on-call experience

Start at the first move in "Designing the on-call experience" and carry it through to the proof.

Step 1 of 2

Designing the on-call experiencewhat the engineer at 3am actually sees

As the manager, you own whether on-call is humane or miserable. The bar is that a single engineer, woken at 3am, can diagnose and act from the runbook and dashboard alone without paging you. Build toward that.

  1. 1Alert on leading indicators. Page on DLQ depth and retry-rate spikes, not only on a success-rate crater that customers have already felt.
  2. 2One runbook per failure mode. "DLQ growing," "provider returning 5xx," "consumer lag rising" - each with the dashboard link, the likely cause and the first action.
  3. 3Dashboards that answer the first question. The on-call's opening question is "what changed?" - put delivery rate, retry rate, DLQ depth and recent deploys on one screen.
  4. 4Clear ownership per failure mode. Every alert routes to a team that can act on it; an alert with no owner is noise that trains people to ignore the pager.