1 min lesson
Self-hosted vs provider-hosted
Use "A mature platform does both and routes between them" to say what you would do next.
Step 1 of 3
Self-hosted vs provider-hostedthe tradeoff that frames the whole role
This is the strategic fork the inference platform lives on. Hosting your own models on rented or owned GPUs gives you control and the best cost-at-scale, at the price of running a serving stack, capacity planning and on-call. Calling a provider's API gives you flexibility and operational simplicity, at the price of per-token margin, rate limits and being exposed to their outages. A mature platform does both and routes between them.
Learn more
Advanced table
Reference table
- Dimension
- Cost at scale
- Self-hosted
- Lowest if utilization is high
- Provider-hosted
- Per-token margin baked in
- Dimension
- Control / tuning
- Self-hosted
- Full (batching, KV paging, quantization)
- Provider-hosted
- Limited to API knobs
- Dimension
- Operational burden
- Self-hosted
- You run serving + on-call + capacity
- Provider-hosted
- Provider runs it
- Dimension
- Flexibility / new models
- Self-hosted
- You must deploy each one
- Provider-hosted
- New models appear behind one API
- Dimension
- Failure exposure
- Self-hosted
- Your fleet, your blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition.
- Provider-hosted
- Their outage degrades you unless you fail over
| Dimension | Self-hosted | Provider-hosted |
|---|---|---|
| Cost at scale | Lowest if utilization is high | Per-token margin baked in |
| Control / tuning | Full (batching, KV paging, quantization) | Limited to API knobs |
| Operational burden | You run serving + on-call + capacity | Provider runs it |
| Flexibility / new models | You must deploy each one | New models appear behind one API |
| Failure exposure | Your fleet, your blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition. | Their outage degrades you unless you fail over |
The honest answer in an interview is "it depends on volume, latency budget and which models" - then reason it.