1 min lesson
The split: thin adapter, fat core
Use "Decide deliberately what lives in the adapter versus the shared core" to say what you would do next.
Step 1 of 3
The split: thin adapter, fat core
Decide deliberately what lives in the adapter versus the shared core. Put too much in the adapter and every provider reimplements retries badly. Put provider quirks in the core and you leak the abstraction. The stack figure above enumerates the exact split - the adapter's per-provider wire work versus the core's owned-once cross-cutting concerns.
Learn more
Full explanation
Streaming passthrough is first-class
Streaming passthrough is first-class
The single biggest gateway mistake is buffering the full response before forwarding it. That destroys time-to-first-token, which is the latency a developer actually feels when text starts appearing in the editor.
- Forward token deltas as they arrive; never wait for the final chunk to start the client stream.
- Keep the gateway's per-request memory flat - proxy the stream, don't accumulate it.
- Tap the stream for metrics (TTFT, inter-token latency, token counts) without blocking forwarding.
- Propagate client cancellation downstream so an abandoned Agent turn stops burning provider tokens.