Skip to lesson
Exit
Evals & Production Reliability1 / 3

1 min lesson

Rollback and kill switches

Rebuild the main list in "Rollback and kill switches", then say what each item changes.

Step 1 of 3

Rollback and kill switches

  • A kill switch that disables the workflow instantly, owned by someone who can pull it at 2am without a deploy.
  • A rollback path for whatever the agent changed - easy when edits go through PRs, which is one more reason to route them that way.
  • A documented “what we do when it misbehaves” that the customer's team has actually read, not just received.
Learn more

Full explanation

Incident response for AI

Incident response for AIa bad output is an incident

  1. 1Triage. Is this one bad PR or is the success rate falling for everyone? Check the dashboard before touching anything.
  2. 2Mitigate. Pull the kill switch or narrow the guardrails to stop the bleeding. Mitigation first, root cause second.
  3. 3Diagnose. Open the trace, find the variable that changed - model snapshot, context, a prompt edit, an upstream timeout.
  4. 4Fix the root cause. Change the prompt, the context, the guardrail or the model, whichever the trace points at.
  5. 5Add an eval. Encode the failing case as a golden task so the gate would have caught it. The incident is only closed when it can't silently recur.
Close the loop

The deliverable of an incident isn't the hotfix, it's the new eval. If the same failure can happen again without tripping a gate, you mitigated a symptom and learned nothing durable. “We add a regression eval for every incident” is a sentence that makes senior engineers exhale.

Learn more

Full explanation

Incident response, the other direction: agents as responders

Incident response, the other direction: agents as respondersCursor’s own dogfood story

Cursor’s automations feature was born from exactly this problem - it started internally so engineers wouldn’t wake at 3am to comb logs. On a PagerDuty trigger, a cloud agent uses the Datadog MCPModel Context Protocol. A standard that lets an AI agent pull in context from outside the repo, like Jira tickets or internal docs. Press Enter for the full definition. to fetch logs, investigate, find root cause and - if severity is low - implement a fix. The on-call engineer opens their laptop to a root cause and a PR ready to merge, not a blank page, which matters when an outage can cost millions per minute.

Governing a responder agent

Automations never merge directly - they always produce a human-reviewed PR, and the person clicking merge is the accountable one.

The one genuinely new attack vector is who can trigger them: an automation anyone in a Slack channel can fire is a path to your code - don’t wire people who shouldn’t touch the repo to a code-touching automation.

Scope tools to least privilege (e.g. a read-only SQL/Databricks connection so the agent can fetch data but never drop a prod table) and pair them with pre-action hooks. Note: cloud agents respect account privacy mode but not legacy privacy mode, since the VM must store code for the run.

Watch out

Don't promise zero failures - it's both false and a bad sign to a sharp interviewer. Promise that failures are bounded, caught fast and turned into evals. Truth-seeking means naming the risk and showing the machinery that contains it.

Learn more

Optional practice

Practice: Rollback and kill switches

QAfter mitigating a bad-output incident, what single step actually closes the loop and why?