1 min lesson

Abuse, DDoS and protecting expensive backends

Walk through the important items in "Abuse, DDoS and protecting expensive backends" and give the practical point of each.

Abuse, DDoS and protecting expensive backendsthe inference path is the crown jewel

DDoS volumetrics get absorbed by Anycast and scrubbing at the edge. The subtler threat is abuse that looks legitimate: a credential-stuffed account or a leaked token driving thousands of real, expensive model calls. Defend the wallet, not just the wire.

Tier limits by cost: a cheap autocomplete endpoint gets a generous bucket; a long-context agent run gets a tight one tied to plan.
Add an anomaly signal on spend velocity per account, not just request count, so a sudden cost spike trips a challenge.
Use challenge flows (a managed bot check) for suspicious clients before you outright block, to avoid false-positive lockouts.
Return honest 429s with a Retry-After hint so well-behaved clients back off instead of retry-storming you.

Learn more

Full explanation

A 429 that tells a good client exactly how to behave

A 429 that tells a good client exactly how to behave. Backoff hints turn a hammer into a queue.

HTTP/1.1 429 Too Many Requests
Retry-After: 8
RateLimit-Limit: 60
RateLimit-Remaining: 0
RateLimit-Reset: 8
Content-Type: application/json

{
  "error": "rate_limited",
  "scope": "per_account",
  "retry_after_seconds": 8
}

Watch out

Don't rate-limit purely on IP for an auth'd developer tool. Whole companies sit behind one egress IP, so an IP cap throttles a paying customer's entire office while a botnet across 10,000 residential IPs sails under it. Limit on the identity that maps to cost and entitlement and keep IP as a coarse backstop.

Say it like this

"I'd put a token-bucket limiter at the edge keyed on account and API key, enforced locally per node with async reconciliation to a shared store so I don't pay a cross-region round trip per request. The expensive inference endpoints get tighter, plan-aware buckets plus a spend-velocity anomaly check, because for us a limit breach is a cost event, not only a load event."

Learn more

Optional practice

Practice: Abuse, DDoS and protecting expensive backends

QYou're rate limiting across 200 edge nodes and a single counter in a shared store is becoming a bottleneck on hot keys. What's a sound approach and what do you give up?

Abuse, DDoS and protecting expensive backends

Step 1 of 2