1 min lesson
De-noise before you count
Walk through "De-noise before you count" in order, then name the proof that tells you it worked.
Step 1 of 2
De-noise before you count
Raw logs over-count failures and under-count truth. Three retries of one user action look like three failures and one success unless you collapse them. Client clock skew makes some latencies negative. Bot and automation traffic has a different distribution than humans. An instrumentation bug can manufacture a regression that never touched a user.
- 1Sessionize. Group events into logical attempts so one user action is one row, not one row per retry.
- 2Dedup retries and timeouts. Collapse a retry chain into a single attempt with a final outcome before you compute failure rate.
- 3Filter non-human traffic. Segment out bots and automation or at least tag them so they never silently inflate a metric.
- 4Sanity-check the clock. Drop or correct negative and absurd durations from client skew before they poison percentiles.
- 5Audit the instrument. When a metric jumps, rule out an instrumentation or schema change before declaring a real regression.
Sessionization changes the denominator
Counting raw log lines answers "how many requests failed." Counting sessionized attempts answers "how many user actions failed," which is what reliability actually means. The two can disagree by a wide margin precisely when retries spike - exactly the moment you most need the right number.