1 min lesson
Separate adapters from shared policy
Decide whether behavior belongs in an adapter or the shared gateway core, then test that boundary.
Step 1 of 2
Separate adapters from shared behavior
The adapter should own provider-specific serialization, streaming and error mapping. The core should own behavior that must stay consistent across providers. Duplicating retries inside adapters makes budgets and failure handling drift. Moving provider quirks into the core makes the shared interface harder to use. Test the boundary with provider-neutral fixtures. Each adapter should turn the same shared request into the expected wire payload, return the same internal stream events and classify equivalent failures the same way. Fault tests should also confirm that retries, timeouts and accounting still come from the core rather than an adapter. If a rule changes policy across providers, it belongs in the core. If it changes only the wire representation, it belongs in the adapter.
Learn more
Full explanation
Stream without buffering
Stream responses as they arrive
Buffering the full response delays the first token until generation finishes. Forward each token delta as it arrives so the editor can show progress immediately.
- Forward token deltas as they arrive. Do not wait for the final chunk to start the client stream.
- Keep per-request memory flat. Proxy the stream instead of accumulating it.
- Tap the stream for TTFT, inter-token latency and token counts without blocking forwarding.
- Propagate client cancellation downstream so an abandoned Agent turn stops burning provider tokens.