Shape traffic before the provider rejects it

Shape traffic before the provider rejects it

Providers publish RPM and TPM limits. If you let traffic blow past them, the provider returns 429s, your retry logic kicks in and you've manufactured a retry storm out of your own success. The fix is to enforce the provider's budget proactively at the gateway.

1. 1Model the provider's budget locally. Mirror each provider's RPM/TPM as a token bucket the gateway owns, refilled at the published rate.
2. 2Admit against the bucket. A request that would exceed the budget is shed or queued briefly at the gateway - never forwarded to be rejected upstream.
3. 3Reserve headroom. Run the bucket below the hard limit so a burst doesn't tip you over and so failover traffic from another provider has room to land.
4. 4Adapt to reality. If the provider starts 429-ing despite your model, treat that as a signal to throttle harder, not to retry into it.