1 min lesson
Always-on and bursty
Show why this lesson detail matters: "A developer in another timezone is always working."
Step 1 of 3
Always-on and burstythe two scale realities here
Auth and the agent backend can't take maintenance windows - a developer in another timezone is always working. That pushes toward multi-region and tested failover. The agent load is also spiky, so capacity planning has to target the burst, not the daily average.
Learn more
Advanced table
Validate before incidents do
- Concern
- Always-on auth
- What to plan for
- Multi-region with automated failover; no single-region dependency
- How you'd validate it
- Game-day a region outage and measure failover time
- Concern
- Bursty agent load
- What to plan for
- Headroom for peak (release day, work-hours spike), backpressure when exceeded
- How you'd validate it
- Load test to 2–3x projected peak; watch queue depth and shed behavior
- Concern
- Capacity vs growth
- What to plan for
- Plan against the growth curve, not last quarter's average
- How you'd validate it
- Track utilization trend; alert before saturation, not after
- Concern
- Unknown failure modes
- What to plan for
- Failures you haven't imagined yet
- How you'd validate it
- Chaos testing - kill workers, inject latency, drop a dependency
| Concern | What to plan for | How you'd validate it |
|---|---|---|
| Always-on auth | Multi-region with automated failover; no single-region dependency | Game-day a region outage and measure failover time |
| Bursty agent load | Headroom for peak (release day, work-hours spike), backpressure when exceeded | Load test to 2–3x projected peak; watch queue depth and shed behavior |
| Capacity vs growth | Plan against the growth curve, not last quarter's average | Track utilization trend; alert before saturation, not after |
| Unknown failure modes | Failures you haven't imagined yet | Chaos testing - kill workers, inject latency, drop a dependency |
Validate before incidents do. A failover you've never tested is a hope, not a plan.
Load tests find where the system saturates; chaos tests find what happens when a dependency dies mid-flight. Both run before the incident. The interview-grade version of "reliable" is being able to say "we game-day our auth failover quarterly and load-test the agent queue to 3x peak" - concrete drills beat adjectives every time.
Don't quote five nines for everything - it signals you've never owned an error budget. Real reliability work is choosing where the bar is lower so velocity can be higher and saying out loud which service gets the strict SLO and why. Uniform perfection is a budget you can't afford and a tell you haven't operated at scale.
Learn more
Optional practice
Practice: Always-on and bursty
QAn interviewer asks: "What's your availability target for the agent backend?" You answer "100% - it's our core feature." Why is that a weak answer and what's stronger?