Streaming, tokens and context limits

A gateway that buffers a full response before sending it has already lost. Streaming, token accounting and cancellation are where inference mechanics become concrete gateway code.

Models emit output token by token, almost always over Server-Sent Events (SSE). The whole point is that the user sees text appear as it's generated. A gateway must pass those chunks through as they arrive, not collect the full answer and forward it at the end - buffering would throw away the streaming UX and inflate perceived latency to total latency.

1. 1Open upstream as a stream. Request SSE from the provider and read the response body incrementally, never await the whole thing into memory.
2. 2Flush each chunk downstream. Forward provider events to the client as they land, preserving event boundaries; do not coalesce or reorder.
3. 3Measure both clocks. Record TTFT at the first token and total latency at stream end - they're different numbers and users feel the first one.
4. 4Propagate completion and errors. Send the terminal event on success and on an upstream error mid-stream, surface it cleanly rather than hanging the connection.
5. 5Wire cancellation through. If the client disconnects or aborts, abort the upstream request so you stop generating (and paying for) tokens nobody will read.

// AbortController lets a client cancel propagate to the provider.
async function proxyStream(req: Request, provider: Provider): Promise<Response> {
  const ac = new AbortController();
  req.signal.addEventListener("abort", () => ac.abort()); // client gone -> stop upstream

  const upstream = await provider.fetch({ signal: ac.signal, stream: true });
  if (!upstream.body) throw new GatewayError("no stream body");

  let firstToken = false;
  const t0 = performance.now();
  const out = upstream.body.pipeThrough(
    new TransformStream({
      transform(chunk, controller) {
        if (!firstToken) { firstToken = true; recordTTFT(performance.now() - t0); }
        controller.enqueue(chunk); // flush immediately, no buffering
      },
    }),
  );
  return new Response(out, { headers: { "content-type": "text/event-stream" } });
}