Skip to lesson
Exit
Evals & Production Reliability1 / 2

1 min lesson

The debugging method

Answer "How do you tell a one-off flake apart from a systemic regression and why does it matter?" Go step by step and end with what you would do.

Step 1 of 2

The debugging method

  1. 1Reproduce. Pin the exact input from the trace. A bug you can't reproduce, you can't fix or verify.
  2. 2Inspect the trace. Read what the agent actually saw - the context, the tool calls, the raw output - instead of guessing.
  3. 3Isolate the variable. Change one thing at a time. Swap the model, then the context, then the prompt and watch which move fixes it.
  4. 4Fix the root cause. Apply the change in the layer the taxonomy points at, not the first layer that's convenient.
  5. 5Add an eval. Turn the reproduction into a golden task so the fix is permanent and measured.
Flake or regression?

Before you spend a day on a fix, ask whether it's one bad roll or a real pattern. Re-run the same input several times: if it fails intermittently, you have non-determinism to bound with validation and retries. If it fails every time, you have a systemic bug to fix at the root. Treating a flake like a regression wastes a day; treating a regression like a flake ships a broken workflow.

Learn more

Full explanation

Which knob to turn

Which knob to turnprompt vs. context vs. model vs. guardrails

Change the prompt
when the instruction was ambiguous or the output format was wrong
Change the context
when the model had the wrong or missing information - usually the real culprit
Change the model
when the task genuinely exceeds capability, after ruling out context and prompt
Change the guardrails
when the output was unsafe or out of bounds and needs hard limits, not better odds

Reach for context and prompt first; they're cheaper and fix more failures than a model swap.