Designing the inference gateway

Every Tab completion, every Agent step, every chat turn in Cursor exits through one door. The inference gateway is that door and the JD describes the job as building a single abstraction over every provider's API so onboarding a new model is a config change, not a code change.

Anthropic, OpenAI, Google and a fleet of self-hosted models each speak a different dialect. Their auth differs, their request and response shapes differ, their streaming framing differs and their error vocabularies disagree about what a 429 even means. The gateway's first job is to swallow that heterogeneity so the rest of the system reasons about one model interface.

Frame it to an interviewer as an adapter pattern with teeth. A thin per-provider adapter translates the wire format. A fat shared core owns everything that should never be reimplemented per provider: timeouts, retries, tracing, token accounting, idempotency.