Skip to lesson
Exit
Observability, Quality, Cost & Security1 / 3

2 min lesson

Compute: stop paying for idle

Recall the main items in "Compute: stop paying for idle", then connect each one to the work.

Step 1 of 3

Compute: stop paying for idlethe biggest dumb-money leak

  • Right-size clusters. Match worker type and count to the job's shuffle and memory profile, not a copy-pasted default that's 4x too big.
  • Autoscale and auto-terminate. Scale workers to the workload and kill idle clusters. An interactive cluster left running overnight is pure waste.
  • Spot for the interruptible. Backfills and stateless batch ride spot or preemptible instances; keep on-demand for the latency-critical tier-1 path.
  • Tune the job. Fix skew, prefer broadcast joins for small dimensions, cache reused datasets and turn on Photon where the workload suits it.
Learn more

Full explanation

Attribution: make teams see their own spend

Attribution: make teams see their own spendyou can't optimize what no one owns

A single platform bill nobody can decompose is a bill nobody will reduce. Tag every cluster and job by team and pipeline, lean on Unity Catalog system tables for usage and surface a per-team breakdown. Showback tells each team what it spent; chargeback bills that spend to its budget. Showback usually comes first, because you earn the right to charge back by running clean, trusted showback.