2 min lesson
Predeclare the kill criterion
Take "Predeclare the kill criterion" step by step, then finish with the result that proves it worked.
Step 1 of 2
Predeclare the kill criteriontruth-seeking, made concrete
Before the run, write down the result that would make you abandon the idea. Deciding the bar in advance defends you against moving the goalposts when you're emotionally invested, which is the whole point of truth-seeking as a discipline rather than a slogan.
- 1State the hypothesis. A specific, falsifiable claim: 'grader G raises live edit-accept rate by >=2pts over the baseline.'
- 2Define the kill criterion. The result that ends the idea: 'if the seed-averaged delta is under 1pt or live impact is flat, I stop.'
- 3Pre-register the eval. Fix the eval set, seeds and metric before looking, so you can't retrofit a win.
- 4Run, then report honestly. Show the spread, name confounds you couldn't rule out and state what you'd do next - including killing it.
"Before running, I wrote the kill criterion: under a 1-point seed-averaged gain or no movement on live accept-rate, I'd drop it. The offline number came in at +1.4 but live was flat, so I traced it - turned out the grader was rewarding longer diffs, not better ones. I killed that grader and reported the negative result."
Volunteer a result you killed. Describing an experiment you abandoned because the data said so, calmly and without spin, is the single most credible demonstration of truth-seeking you can give. Cursor would rather hire someone who killed their own pet idea than someone who's never reported a negative result.