1 min lesson
Measure common and slow requests
Use the median, tail percentiles and a measured usefulness deadline to find late results.
Step 1 of 2
Treat tail latency as product behavior
A stable median can hide a smaller group of suggestions that arrive after they are useful. Set the usefulness cutoff from product telemetry and user studies, then measure the share of results that cross it by client, region and route.
- Track the median and tail percentiles. A complete SLO needs both common-case speed and a bound on slow requests.
- Under pressure, use an approved faster route or skip work that can no longer finish before the usefulness deadline.
- Cancel work tied to an older editor state and do not render a result after its request is stale.
Watch out
Do not copy a latency target from an old benchmark or another product. Measure the current user-visible deadline and the full distribution for each route. Recheck it after client, network or model changes.
Learn more
Optional practice
Check a tail latency policy
QWhy should a code-completion service track tail latency as well as the median?