1 min lesson
New bug or regression? Correlate with the timeline
Explain your answer to "Why is it risky to correlate a spike in reports only against app release timestamps?" Add one concrete detail from the lesson.
Step 1 of 2
New bug or regression? Correlate with the timelineCursor ships fast, so regressions are constant
The single most useful question about a spike is "what changed?" Overlay the cluster's first-seen timestamps on two timelines: app releases and model-version swaps. A cluster that appears the hour a new build rolls out is a regression and jumps the queue. A cluster smeared evenly across weeks is a long-standing issue you're only now seeing clearly.
- Pattern in the cluster
- First-seen clusters tightly at a release timestamp
- Likely diagnosis
- Release regression
- Action
- Escalate as regression; flag the suspect build for bisect
- Pattern in the cluster
- First-seen aligns with a model-version swap
- Likely diagnosis
- Model regression: behavior changed silently
- Action
- Pin model version in the report; compare old vs new on a fixed prompt
- Pattern in the cluster
- Steady trickle across many versions
- Likely diagnosis
- Long-standing latent bug
- Action
- Prioritize on volume × ARR; no rollback urgency
- Pattern in the cluster
- Sharp spike, then decay, no release nearby
- Likely diagnosis
- External cause: provider outage or viral thread
- Action
- Confirm upstream; communicate status, don't chase a code fix
| Pattern in the cluster | Likely diagnosis | Action |
|---|---|---|
| First-seen clusters tightly at a release timestamp | Release regression | Escalate as regression; flag the suspect build for bisect |
| First-seen aligns with a model-version swap | Model regression: behavior changed silently | Pin model version in the report; compare old vs new on a fixed prompt |
| Steady trickle across many versions | Long-standing latent bug | Prioritize on volume × ARR; no rollback urgency |
| Sharp spike, then decay, no release nearby | External cause: provider outage or viral thread | Confirm upstream; communicate status, don't chase a code fix |
A model swap is an invisible deploy. The app version can be identical while behavior, diff quality or latency shifts because the backend now routes to a different model. If you only correlate against app releases, you'll misclassify a whole class of regression as a mysterious new bug. Always log the model version alongside the build.
Learn more
Full explanation
Full explanation
"Before I rank anything, I'd collapse these 23 tickets. They come down to 4 clusters. Cluster A is 14 reports, all S1 Tab failures, and 11 of them share one error signature while the other three match on symptom and version. Their first-seen times all land within an hour of one build rolling out, so that's a regression. Cluster B is 5 reports of slow indexing spread across versions: a long-standing S2. Cluster C is 3 reports of one misaligned icon: S3. Cluster D is a single report of an Agent edit overwriting unsaved files."
"The order is D, A, B, C. D goes first because it's S0 data loss and data loss has no quorum, even with one reporter. A is next: a fresh regression hitting 14 people with no workaround. B and C wait for the backlog."