1 min lesson
Priority and QoS classes
Use the lesson to respond to this: "Under a traffic spike, why is shedding excess requests often better than queueing them, especially for Tab completions?" Keep the answer plain.
Step 1 of 2
Priority and QoS classes
Not all traffic is equal. A Tab completion has a sub-100ms budget and a user waiting on a keystroke. A background re-index or a bulk eval job can wait. Under contention, the latency-critical class must win.
- Class
- Interactive / latency-critical
- Example surface
- Tab, ⌘K
- Under contention
- Protected - shed last, never queued past budget
- Class
- Interactive / tolerant
- Example surface
- Agent, chat
- Under contention
- Served, but may degrade to a fallback model
- Class
- Bulk / background
- Example surface
- Re-index, batch eval
- Under contention
- Shed first, deferred or rate-limited hard
| Class | Example surface | Under contention |
|---|---|---|
| Interactive / latency-critical | Tab, ⌘K | Protected - shed last, never queued past budget |
| Interactive / tolerant | Agent, chat | Served, but may degrade to a fallback model |
| Bulk / background | Re-index, batch eval | Shed first, deferred or rate-limited hard |
Fair queuing within a class; strict priority across classes - Tab does not wait behind a batch job.
Shedding a request cleanly is kinder than queueing it past its usefulness. A sub-100ms Tab request that sits in a queue for 400ms is already worthless when it dequeues - you spent capacity to deliver a stale answer the user has stopped waiting for. A fast, clear rejection lets the client back off intelligently; a slow death just guarantees a retry on top of the original load.
A shed request must return a non-cascading error. If load shedding produces a generic 500, clients treat it as retryable, retry immediately and you've built a feedback loop that amplifies the spike. Use a distinct, explicitly-retryable signal with a backoff hint so the client's reaction relieves pressure instead of adding to it.