Mixed Methods & Quant Literacy

Anyone can run interviews. The methods panel is checking whether your qual would survive a skeptical staff engineer reading the transcript next to your claim.

Cursor's users are professional developers who can smell a leading question and will tell you their workaround instead of their real problem. Rigor here is not academic ceremony. It is the chain you can show from a raw quote to a coded theme to the recommendation you put in front of a PM, with the bias-control choices visible at each step.

Designing the interview guidethe work happens before the call

A guide is a hypothesis about what you don't know, written as questions you can't accidentally answer for the participant. Two failure modes dominate: questions that telegraph the answer and questions that ask people to predict their own behavior.

Once you have a behavior, ladder to the why. Each answer earns one more “what made that matter to you?” until you hit a value or a constraint, not another feature request.

Moderation disciplinewhat you do on the call

1. 1Silence is a tool. After a participant stops, count to three before speaking. The richest material lands in the pause they fill to escape the quiet.
2. 2Follow the energy, not the script. When a developer leans in about reviewing agent diffs, abandon question seven and chase it. The guide is a safety net, not a railroad.
3. 3Mirror, don't lead. Reflect their words back (“you said it felt risky”) rather than supplying your own (“so it was scary?”). Your vocabulary contaminates theirs.
4. 4Park your hypothesis. If you walked in believing latency is the problem, you will hear latency everywhere. Note the urge, then deliberately probe the cases that would prove you wrong.

Analysis you can auditfrom transcript to claim

Thematic analysis is where qual earns or loses its credibility. The point is a transparent trail, so anyone can trace a theme back to the quotes underneath it.

• Code close to the data first. Tag what was actually said before you invent higher-level categories, so your themes are grounded rather than imposed.
• Affinity-map the codes. Cluster related codes into themes; the cluster sizes give you a rough sense of weight without pretending qual produces percentages.
• Track saturation honestly. When three or four straight interviews stop producing new codes, you've likely saturated that question. Name the number; don't claim saturation at n=4 if codes are still arriving.
• Keep a quote ledger. Every claim in the readout links to two or three verbatim quotes. That ledger is what makes the finding defensible to an engineer.

A survey looks like the easy method and is the easiest to get wrong. A badly worded item produces clean-looking numbers that mean nothing and nobody downstream can tell.

At Cursor a survey's job is usually to size something qual already found or to track a metric over time across a developer base too large to interview. That only works if the questions are unbiased and the sample represents who you're claiming to speak for.

Writing items that measure one thingthe question is the instrument

Pick scales deliberately. A balanced 5- or 7-point Likert with a neutral midpoint suits agreement and satisfaction; for likelihood-to-recommend you inherit the 0–10 NPS convention. Match the scale to the construct and keep every item's scale consistent so respondents don't have to re-learn the rules mid-survey.

Where the bias hideserrors that survive a clean-looking dataset

Sampling and how confident you getwho, how many, how sure

Sample size is a precision dial, not a magic threshold. Bigger samples narrow the margin of error with diminishing returns and they never fix a biased frame. Decide who you need first, then how many.

Closed plus open, analyzed together

• Pair a number with a reason. Follow a rating item with one open “what drove that score?” to recover the why without a full interview.
• Code the verbatims like qual. Tag open responses into themes; the closed item tells you how many, the open text tells you what they mean.
• Read them against each other. When a satisfaction score is high but verbatims are full of workarounds, the open text is usually closer to the truth.

You won't own the stats stack at Cursor, but you have to be fluent enough to read an experiment, challenge a metric and know when the data team's number doesn't answer the research question.

Quant literacy is what separates a researcher who triangulates from one who just runs interviews. The bar is partnership: you frame the question and interpret the result, the data team owns the pipeline and the test.

A/B testing, enough to be dangerousthe skeleton you should recognize

1. 1Hypothesis. A directional, falsifiable claim: “showing a diff preview before applying agent edits will raise acceptance without raising reverts.”
2. 2Control vs. treatment. Randomly assign comparable groups so the only systematic difference is the change you're testing.
3. 3Primary metric + guardrails. One metric you expect to move, plus a few you refuse to harm (reverts, error rate, retention).
4. 4Significance and power. Decide the smallest effect worth shipping and the sample needed to detect it before launch, then read results once at the planned end.

Reading product analyticswhat the logs tell you

Metrics that matter for an AI editorCursor-flavored fluency

• Acceptance rate of suggestions - share of completions or agent edits a developer keeps. High value, but acceptance isn't correctness; pair it with downstream signals.
• Time-to-first-value - how fast a new user gets a genuinely useful result, a leading indicator for activation and retention.
• Retained daily use - does Cursor become part of the everyday loop, the strongest signal that the product earns its place in a developer's day.
• Revert / undo rate - a guardrail; suggestions accepted and then ripped back out are a quiet quality problem acceptance rate alone hides.

This is the line in the job description made real: combine qualitative research with analytics, surveys and experiments. Triangulation is the senior skill and it's what most candidates do shallowly.

Each method has a blind spot the others cover. The craft is not running all four; it's using each for what it's best at and stitching them into one decision-ready story with honest confidence attached.

Sequence the methodsorder is part of the design

1. 1Interview to discover. A handful of sessions surface the real friction and the developers' own language for it.
2. 2Survey to size. Field a short instrument to learn how widespread the friction is and which segments feel it most.
3. 3Analytics to ground. Check whether behavior in the logs matches what people said and quantify the funnel where it bites.
4. 4Experiment to confirm. Ship a change behind an A/B test to prove the fix actually moves the metric, not just sentiment.

When signals conflictthe part that reveals seniority

Conflicting data is the normal case, not a failure. Resolve it explicitly by deciding which source is authoritative for which kind of claim, then say so out loud.

The methods panel includes researchers who've been burned by every one of these. They're not testing whether you can recite definitions; they're watching whether you catch the trap when it's hiding in a plausible-sounding result.

The traps and how to name them

The two that catch the most people

Significance versus practical importance is the most common stumble, because a huge sample makes almost any difference statistically significant. The question is never just “is it real?” but “is it big enough to act on?”

Confounds in observational data are the subtler trap. The agent-and-retention example feels causal and isn't: power users select into both, so the relationship can be entirely selection. The only clean fix is randomization, which is why you reach for an experiment when the stakes justify it.