1 min lesson
Rate limiting and abuse defense
Use two rows in "Rate limiting and abuse defense" to state the practical decision rules.
Step 1 of 2
Rate limiting is where infra meets unit economics. Every model call has a hard dollar cost, so a runaway script or a stolen API key isn't just a reliability problem - it's a bill.
The interviewer wants to see that you can pick an algorithm for a reason, run it across many edge nodes without a hot key meltdown and degrade kindly when a real user brushes the limit. Start with the algorithms.
- Algorithm
- Fixed window
- Burst behavior
- Allows a 2x spike at the window boundary
- Tradeoff
- Simplest and cheapest; unfair edge effect lets bursts through
- Algorithm
- Sliding window
- Burst behavior
- Smooths the boundary spike
- Tradeoff
- Fairer; needs per-request timestamp state or a weighted approximation
- Algorithm
- Token bucket
- Burst behavior
- Allows controlled bursts up to bucket size, then steady refill
- Tradeoff
- Best UX for bursty clients; two params (rate, burst) to tune
- Algorithm
- Leaky bucket
- Burst behavior
- No bursts; constant drain rate
- Tradeoff
- Protects a fragile backend with fixed throughput; can feel sluggish
| Algorithm | Burst behavior | Tradeoff |
|---|---|---|
| Fixed window | Allows a 2x spike at the window boundary | Simplest and cheapest; unfair edge effect lets bursts through |
| Sliding window | Smooths the boundary spike | Fairer; needs per-request timestamp state or a weighted approximation |
| Token bucket | Allows controlled bursts up to bucket size, then steady refill | Best UX for bursty clients; two params (rate, burst) to tune |
| Leaky bucket | No bursts; constant drain rate | Protects a fragile backend with fixed throughput; can feel sluggish |
Token bucket is the usual default for user-facing APIs: it forgives a quick burst, then enforces a sustained rate.
For Cursor, token bucket fits the human pattern: a developer fires several agent requests in a flurry, then goes quiet to read the diff. You want to allow the flurry and cap the sustained rate, which is exactly the burst + refill shape.