2 min lesson
Evaluate retrieval on its own
Take this situation. "Your agent keeps producing incorrect edits on a large repo. Before you touch the prompt or swap models, what should you check first and why?" Lead with your decision, then add the reason.
Step 1 of 2
Evaluate retrieval on its owndon't blame the model for a bad fetch
If you only measure final edits, you can't tell whether a bad result came from the model or from never giving it the right file. Score retrieval independently before you judge generation, starting with recall, the share of must-have files that came back.
- Build a set of queries with the chunks a human says should come back.
- Measure recall: did the must-have files land in the top-k results? Top-k means the k best-ranked hits you actually pass to the model.
- Track precision, the share of returned chunks that are relevant, so you're not flooding the window with near-misses that cost tokens.
- When generation is wrong, check the retrieved set first. Often the file the model needed simply wasn't there.
"Say I write 20 test queries, each with the file a human says must come back, and the right file shows up in the top 5 for 13 of them. That's 65% recall, so retrieval is missing the file and I fix chunking and the search blend before I touch the prompt. Those numbers are made up to show the method."
Cursor indexes the codebase so the editor and Agent can pull relevant files into context automatically.
When you deploy a workflow, you're building on that retrieval layer, so "did we retrieve the right code" is a question you can answer with its index, not one you reinvent.
Learn more
Full explanation
Full explanation
If a design question reaches "the agent gives wrong answers on a big repo," split the problem out loud: is retrieval missing the file or is generation fumbling a file it had?
Proposing a retrieval eval before touching the prompt reads as production experience.