1 min lesson

Concurrency, streaming and resilience patterns

Tell someone how to act on this idea: "You will not build the whole platform in a screen."

These five primitives are the reusable parts of an inference gateway. Interviewers ask for one in code and then probe whether you understand how it composes with the others under load.

You will not build the whole platform in a screen. You will be asked to implement one piece cleanly and reason about its interactions: what happens when a retry fires inside a hedge or when a timeout cancels a request that already mutated state.

Learn more

Advanced table

Each is small alone; the interview is in how they interact

Primitive: Timeout + cancellation
What it does: Bound every upstream call; abort downstream work when the client disconnects
The trap interviewers probe: Leaking a hung connection or not propagating the cancel - work keeps burning GPU after nobody's waiting

Primitive: Backoff + jitter
What it does: Space out retries so failures don't synchronize into a storm
The trap interviewers probe: Forgetting jitter, so all clients retry on the same tick and re-DDoS the recovering provider

Primitive: Hedged request
What it does: Send a backup after a delay; take the first success, cancel the loser
The trap interviewers probe: Unbounded duplicate cost - hedging too eagerly doubles real provider spend

Primitive: Bounded concurrency
What it does: Cap in-flight work with a semaphore/queue; apply backpressure
The trap interviewers probe: Unbounded fan-out that melts the fleet instead of shedding load

Primitive: Idempotency key
What it does: Make a retried/hedged call safe to execute more than once
The trap interviewers probe: Double-charging or double-applying a side effect when two attempts both land

Primitive	What it does	The trap interviewers probe
Timeout + cancellation	Bound every upstream call; abort downstream work when the client disconnects	Leaking a hung connection or not propagating the cancel - work keeps burning GPU after nobody's waiting
Backoff + jitter	Space out retries so failures don't synchronize into a storm	Forgetting jitter, so all clients retry on the same tick and re-DDoS the recovering provider
Hedged request	Send a backup after a delay; take the first success, cancel the loser	Unbounded duplicate cost - hedging too eagerly doubles real provider spend
Bounded concurrency	Cap in-flight work with a semaphore/queue; apply backpressure	Unbounded fan-out that melts the fleet instead of shedding load
Idempotency key	Make a retried/hedged call safe to execute more than once	Double-charging or double-applying a side effect when two attempts both land

Each is small alone; the interview is in how they interact.

Learn more

Full explanation

Backoff with jitter and a budget

Backoff with jitter and a budgetthe storm-avoidance primitive

Full jitter plus a retry budget - caps both synchronized retries and total attempts.ts

async function withRetry<T>(
  fn: (signal: AbortSignal) => Promise<T>,
  opts: { maxAttempts: number; baseMs: number; capMs: number; signal: AbortSignal },
): Promise<T> {
  let attempt = 0;
  for (;;) {
    try {
      return await fn(opts.signal);
    } catch (err) {
      attempt++;
      if (attempt >= opts.maxAttempts || opts.signal.aborted) throw err;
      const ceil = Math.min(opts.capMs, opts.baseMs * 2 ** attempt);
      const delay = Math.random() * ceil; // full jitter
      await sleep(delay, opts.signal);
    }
  }
}

Two details earn the credit. Full jitter (random() * ceil, not ceil + random()) is what actually de-synchronizes a thundering herd. A bounded maxAttempts is the retry budget that stops a dead provider from amplifying load forever.

Concurrency, streaming and resilience patterns

Step 1 of 3

Each is small alone; the interview is in how they interact

Backoff with jitter and a budget

Backoff with jitter and a budgetthe storm-avoidance primitive