1 min lesson
Model regression vs. harness regression
Answer this as if it were happening now: "The agent-run success metric drops sharply. Acceptance rate fell but latency and tool-call error rates are flat. What does this point to and who should be paged?" Say what supports your choice.
Step 1 of 2
Model regression vs. harness regressionpage the right team
When the success metric drops, the most valuable thing you can do is attribute it. A model regression means the outputs got worse. A harness regression means the loop got slower or flakier while the model is fine. They page different teams and conflating them wastes the on-call.
- Symptom
- Acceptance rate drops, latency flat
- Likely model regression
- yes - output quality fell
- Likely harness regression
- no
- Symptom
- Latency and timeout rate spike, acceptance flat
- Likely model regression
- no
- Likely harness regression
- yes - loop or tools degraded
- Symptom
- Tool-call error rate jumps
- Likely model regression
- rarely
- Likely harness regression
- yes - harness or integration
- Symptom
- Drop isolated to one model version
- Likely model regression
- yes - model rollout
- Likely harness regression
- no
| Symptom | Likely model regression | Likely harness regression |
|---|---|---|
| Acceptance rate drops, latency flat | yes - output quality fell | no |
| Latency and timeout rate spike, acceptance flat | no | yes - loop or tools degraded |
| Tool-call error rate jumps | rarely | yes - harness or integration |
| Drop isolated to one model version | yes - model rollout | no |
Hold one axis fixed to separate output quality from loop health.
A success metric that rewards short, safe answers will quietly train the product to be useless. If "no undo within 10s" defines success, the agent learns to make timid edits no one bothers to revert. Always ask: what would a lazy optimizer do to my metric and does that path make the product worse?