2 min lesson
Tokens are not characters and they're not portable
Talk this through in your own words: "A user cancels a long Agent generation. What must the gateway do and why is it more than a UX nicety?" Finish with the next move.
Step 1 of 2
Tokens are not characters and they're not portablecounting is per-provider
Tokenization differs by model and provider - the same string is a different token count under different tokenizers. Token counts drive three things you can't fudge: cost estimates (pricing is per token), context-limit checks (does this request fit?) and rate-limit accounting (TPM budgets). A gateway that estimates tokens with the wrong tokenizer will misprice requests and either reject valid ones or let oversized ones through.
- Cost estimation
- Per-token input/output pricing - wrong count, wrong bill prediction
- Context-limit checks
- Reject or down-route before the provider 400s on an oversized prompt
- Rate-limit accounting
- TPM (tokens-per-minute) budgets need real counts, not character heuristics
Context windows are finite, so the gateway needs a graceful answer when a request won't fit. Two honest options: reject early with a clear error or down-route to a model with a bigger window (and a different price). Silently truncating the prompt is the wrong move - it changes the user's request without telling them.
When a user hits stop on an Agent run, the gateway must propagate that cancellation upstream. If it doesn't, the provider keeps decoding tokens the user will never see and you keep paying output-token prices for them. "Client abort fans out to an AbortController on the upstream request" is the pattern - and it's both a UX and a real-dollars concern at Cursor's volume.
Counting tokens by dividing characters by four is fine for a rough log line and dangerous for billing or admission. For anything load-bearing, use the model's actual tokenizer or the provider's usage numbers, because the heuristic drifts hard on code, non-English text and long identifiers.
Learn more
Optional practice
Practice: Tokens are not characters and they're not portable
QWhy can't a gateway rely on a single global "characters ÷ 4" rule to count tokens for billing and admission?