Contamination: the benchmark is in the training data

Contamination: the benchmark is in the training datathe credibility killer

Public benchmarks leak. The GitHub issues behind SWE-bench, the HumanEval prompts, the Stack Overflow answers - they're in the pretraining corpus. A model can score high by having memorized the fix, not by being able to solve the task. A benchmark gain on contaminated data measures recall, not capability.

• Prefer private, freshly-authored eval tasks the model could not have seen - newer than the training cutoff or written internally.
• Distribution-match the eval to what Cursor users actually ask, not to whatever benchmark is convenient to score.
• Run contamination checks: look for verbatim recall, test whether perturbing the problem tanks the score.
• Refresh evals over time - a private eval becomes a contaminated one once it's been trained on for a few cycles.