Skip to lesson
Exit
Automation, IaC & Zero-Trust Engineering1 / 2

2 min lesson

Runbooks and on-call for IT

Answer this: "Why is it a strong move to define an SLO like 'access is fully revoked within 15 minutes of the HRIS termination event, 99% of the time,' and what should happen when it's missed?"

Step 1 of 2

Runbooks and on-call for ITwhen SSO or offboarding breaks at 5pm Friday

  1. 1Detect. An alert fires on the symptom (provisioning failed, SSOSingle Sign-On. One company login (usually via SAML or OIDC) instead of a separate password per tool. Press Enter for the full definition. error rate up) with context: which system, which user, what changed.
  2. 2Triage and contain. If SSOSingle Sign-On. One company login (usually via SAML or OIDC) instead of a separate password per tool. Press Enter for the full definition. is down org-wide, that's a sev-1; if offboarding stalled, manually revoke the highest-risk access first while you fix the pipeline.
  3. 3Follow the runbook. Each critical system has a written runbook: how to detect, the manual fallback, who to escalate to and the rollback step.
  4. 4Restore and verify. Confirm the fix with the same check that should have caught it - re-run the reconcile loop and confirm convergence.
  5. 5Postmortem, blamelessly. Find the systemic cause, then convert the fix into automation or a new alert so this exact failure can't recur silently.
A manual fix is an unfinished fix

The engineer's-mindset tell in a postmortem: you don't close it by saying "I'll remember to check that next time." You close it by adding the alert that would have caught it earlier or the reconcile step that would have self-healed it. Every incident becomes a permanent improvement to the system, which is how a tiny IT team keeps scaling with a hypergrowth company without drowning in toil.

Interview move

When asked about reliability or an incident, answer in production-engineering vocabulary: SLO, alert on symptoms, runbook, blameless postmortem, turn-the-fix-into-automation. Anchor it on offboarding latency as an SLO - "access revoked within 15 minutes of the HRIS event, paged if missed" - because it shows you think of IT as a measured service and you grasp that offboarding is a security control, not a courtesy.