1 min lesson
Evaluate retrieval on its own
Take this situation: "Your agent keeps producing incorrect edits on a large repo. Before you touch the prompt or swap models, what should you check first and why?" Lead with your decision, then add the reason.
Step 1 of 2
Evaluate retrieval on its owndon’t blame the model for a bad fetch
If you only measure final edits, you can’t tell whether a bad result came from the model or from never giving it the right file. Score retrieval independently before you judge generation.
- Build a set of queries with the chunks a human says should come back.
- Measure recall: did the must-have files land in the top-k results?
- Track precision so you’re not flooding the window with near-misses that cost tokens.
- When generation is wrong, check the retrieved set first - often the file the model needed simply wasn’t there.
Cursor indexes the codebase so the editor and Agent can pull relevant files into context automatically.
When you deploy a workflow, you’re building on that retrieval layer, so “did we retrieve the right code” is a question you can answer with its index, not one you reinvent.
If a design question reaches “the agent gives wrong answers on a big repo,” split the problem out loud: is retrieval missing the file or is generation fumbling a file it had?
Proposing a retrieval eval before touching the prompt reads as production experience.