1 min lesson
Novelty, primacy and pre-launch evidence
For each case in "Novelty, primacy and pre-launch evidence", name the signal and the response you would use.
Step 1 of 2
Novelty, primacy and pre-launch evidenceshort tests mislead with developer tools
Developers react to anything new, up or down, then settle
A 3-day spike can reverse by week 3
Plan a long-running holdout to read the durable effect
Run the change silently or on 1% before a real test
Catches crashes and gross regressions early
Confounded selection, so treat it as a smoke test not proof
If a prompt says "we changed the shared retrieval index, how would you measure it?", the strong answer names the leak first: a user-level split won't isolate a shared resource, so I'd run a switchback or randomize at the cluster boundary, then cluster SEs at the time bucket. Spotting the interference before proposing a design is what separates the answers.
Don't declare a win on day 3 of a developer-tool experiment. Novelty cuts both ways: a new behavior can spike usage that decays or annoy power users who later adapt. If you can't run long, carve out a small long-term holdout so you can read the steady-state effect after the novelty washes out.