Skip to lesson
Exit
AI editor systems and design1 / 2

1 min lesson

Route requests by measured difficulty

Choose an eligible model by measured quality, latency and cost, then test the routing policy.

Step 1 of 2

Control cost with routing

A routing policy can choose among eligible models by expected quality, latency and cost. It needs measured outcomes because request difficulty and model performance change over time.

  1. 1Start with the lowest-cost route that meets the request's capability, policy and measured quality requirements.
  2. 2Estimate difficulty with features that can be evaluated, such as task type, context size and prior route outcomes.
  3. 3Escalate only when the expected quality gain justifies the added latency and cost for that product interaction.
  4. 4Cache only equivalent requests, include policy-relevant inputs in the key and record the hit rate and stale-result rate.
Test the routing policy

Compare the policy with fixed-model baselines in an offline replay and a bounded online experiment. Track usefulness, keep rate, p95 and p99 latency, cost per request and fallback rate. Stop the rollout if a cost gain comes with lower retained-code quality or a worse latency tail.