Skip to lesson
Exit
Routing, failover and backpressure1 / 2

1 min lesson

Onboard a model through configuration

Validate and roll out a model configuration without changing callers or hiding capability gaps.

Step 1 of 2

Adding a model is a config change

The job description names model onboarding by configuration as a design goal. A model that uses an existing provider API can have a declarative entry for its provider, identifier, limits, timeout, price and fallback. Validate the model identifier and declared capabilities before enabling the route. Reject fallback cycles and fallbacks that lack a required capability, context size or placement rule. Check limits, timeout and price fields before distributing the configuration. Test the resolved route with no live traffic, then enable a small share and keep a fast removal control. Give each configuration a version and audit record so a request can be tied to its routing policy and a rollback can be traced.

Learn more

Full explanation

Review an illustrative model configuration

Illustrative model configuration that reuses an existing adapter
models:
  tab-model-a:
    provider: provider_a
    upstream_id: model-a-2026
    surfaces: [tab]
    limits: { rpm: 4000, tpm: 2_000_000 }
    timeout_ms: 800
    price_per_mtok: { in: 3.00, out: 15.00 }
    fallback: tab-model-b
Interview move

Separate two cases. A model on an existing provider uses configuration. A provider with a different API needs an adapter that implements the shared interface, plus its configuration. Explain which fields the adapter translates and which policies remain in the core.

Watch out

Do not hide a provider difference that changes caller behavior. If tool calls or stop sequences behave differently, represent that difference in the typed contract and test it. A false equivalence creates provider-specific production bugs.

Learn more

Optional practice

Place retry policy

QWhere should retry logic live in the gateway, in each provider adapter or in the shared core, and why?