1 min lesson
For nondeterministic LLM behavior
Answer "Why is running a flaky agent bug 20 times more useful than reproducing it once and what does the result give engineering?" Use one lesson detail to support it.
Step 1 of 2
For nondeterministic LLM behaviorrates, not anecdotes
An LLM can give a different answer to the same prompt. One bad output proves nothing - it could be the tail of a distribution. Run the same scenario many times and report the rate. "The agent drops the open file from context in 6 of 20 runs" is a bug an engineer can chase; "it did something weird once" is not.
A scratch harness to turn a flaky agent bug into a measurable rate.
# Fixed inputs: same repo snapshot, same prompt, pinned model.
# Loop the scenario, classify each run, report the frequency.
for i in $(seq 1 20); do
run_agent --repo ./repro-fixture --prompt "$PROMPT" --model auto \
>> runs/$i.log
done
grep -l "overwrote unrelated function" runs/*.log | wc -l # -> failures / 20When you can, build a small scratch repo or script that recreates the trigger deterministically. A fixed-input fixture is the cleanest gift you can hand engineering, because it removes their setup cost entirely.