1 min lesson
Avoiding thrash
Rebuild the parts of "Avoiding thrash", then say why each one matters.
Step 1 of 2
Avoiding thrashthe feedback-loop failure
An autoscaler that reacts too fast oscillates: it scales up on a brief spike, the metric drops, it scales down, the load returns and it scales up again - churning pods and nodes while serving worse than a steady fleet would. Stabilization windows and asymmetric up/down behavior tame it.
- Scale-up fast, scale-down slow
- React quickly to protect users; remove capacity conservatively to avoid yanking it back.
- Stabilization window
- Require the signal to hold for a window before acting, so transient blips don't trigger scaling.
- Sane min/max
- A min that absorbs normal variance and a max that caps cost and protects downstream dependencies.
- Protect dependencies
- Cap max so a scale-up storm can't overwhelm a database or model backend that can't scale with you.
Tie the whole answer to cost-vs-reliability, which is the judgment Cursor's infra loop tests. "I'd scale pods on p99 latency, not CPU, run Karpenter for cheap just-in-time nodes, hold a small headroom buffer so spikes don't wait on boot time and use a stabilization window plus a hard max so we don't thrash or stampede the model backend." That sentence shows you optimize spend and reliability together, not one at the other's expense.
Learn more
Optional practice
Practice: Avoiding thrash
QYour service is CPU-light but calls a slow model backend. Under load, p99 latency climbs and the request queue grows, yet CPU stays near 40% so the HPA never scales up. How do you fix the autoscaling and what do you watch for?