Skip to lesson
Exit
Building AI-Native Workflows1 / 3

2 min lesson

Streaming and backpressure

Walk through the important items in "Streaming and backpressure" and give the practical point of each.

Step 1 of 3

Streaming and backpressuremake partial results usable

  • Stream tokens as they generate so the user sees motion in hundreds of milliseconds instead of waiting for the whole response.
  • Make partial output usable: render the diff as it forms so a reviewer can start reading before it finishes.
  • Apply backpressure when the consumer is slower than the stream, so a fast model can’t flood a slow UI or a downstream tool.
Stream and apply backpressure so the consumer sets the pace.
async def stream_edit(req):
    async for token in model.stream(req):       # tokens as they arrive
        await ui.send(token)                    # render incrementally
        if ui.buffer_full():                    # consumer can't keep up
            await ui.drain()                    # wait before pulling more
Learn more

Full explanation

Match the model to the task

Match the model to the taskdon’t reach for the biggest by reflex

Always using the largest model is the most common waste in production LLM systems. Route by difficulty: a small model handles the easy bulk, a large model handles the hard tail.

Task
Rename a symbol, format, mechanical edit
Tier
Small / fast
Why
Deterministic enough that a small model nails it cheaply.
Task
Localized bug fix with a failing test
Tier
Mid
Why
Needs reasoning but is well-scoped by the test.
Task
Cross-cutting refactor or migration plan
Tier
Large
Why
Wide context and harder reasoning justify the cost.

Tiering by task is the single biggest cost lever in most workflows.