2 min lesson
New bug or regression? Correlate with the timeline
Explain your answer to "Why is it risky to correlate a spike in reports only against app release timestamps?" Add one concrete detail from the lesson.
Step 1 of 2
New bug or regression? Correlate with the timelineCursor ships fast, so regressions are constant
The single most useful question about a spike is “what changed?” Overlay the cluster's first-seen timestamps on two timelines: app releases and model-version swaps. A cluster that appears the hour a new build rolls out is a regression and jumps the queue. A cluster smeared evenly across weeks is a long-standing issue you're only now seeing clearly.
- Pattern in the cluster
- First-seen clusters tightly at a release timestamp
- Likely diagnosis
- Release regression
- Action
- Escalate as regression; flag the suspect build for bisect
- Pattern in the cluster
- First-seen aligns with a model-version swap
- Likely diagnosis
- Model regression - behavior changed silently
- Action
- Pin model version in the report; compare old vs new on a fixed prompt
- Pattern in the cluster
- Steady trickle across many versions
- Likely diagnosis
- Long-standing latent bug
- Action
- Prioritize on volume × ARR; no rollback urgency
- Pattern in the cluster
- Sharp spike, then decay, no release nearby
- Likely diagnosis
- External cause (provider outage, viral thread)
- Action
- Confirm upstream; communicate status, don't chase a code fix
| Pattern in the cluster | Likely diagnosis | Action |
|---|---|---|
| First-seen clusters tightly at a release timestamp | Release regression | Escalate as regression; flag the suspect build for bisect |
| First-seen aligns with a model-version swap | Model regression - behavior changed silently | Pin model version in the report; compare old vs new on a fixed prompt |
| Steady trickle across many versions | Long-standing latent bug | Prioritize on volume × ARR; no rollback urgency |
| Sharp spike, then decay, no release nearby | External cause (provider outage, viral thread) | Confirm upstream; communicate status, don't chase a code fix |
A model swap is an invisible deploy. The app version can be identical while behavior, diff quality or latency shifts because the backend now routes to a different model. If you only correlate against app releases, you'll misclassify a whole class of regression as a mysterious new bug. Always log the model version alongside the build.
On the messy-bug-batch take-home, narrate the collapse: “These 23 tickets are 4 clusters. Cluster A is 14 reports, all on build 0.42, all S1 Tab failures - that's a regression and it leads.” Showing the firehose-to-shortlist compression, with a count and a release correlation, is exactly the prioritization-judgment artifact the stage grades.