Skip to lesson
Exit
Observability, Quality, Cost & Security1 / 3

2 min lesson

Compute: stop paying for idle

Recall the main items in "Compute: stop paying for idle", then connect each one to the work.

Step 1 of 3

Compute: stop paying for idlethe biggest dumb-money leak

  • Right-size clusters. Match worker type and count to the job's shuffle and memory profile, not a copy-pasted default that's 4x too big.
  • Autoscale and auto-terminate. Scale workers to the workload and kill idle clusters - an interactive cluster left running overnight is pure waste.
  • Spot for the interruptible. Backfills and stateless batch ride spot/preemptible instances; keep on-demand for the latency-critical tier-1 path.
  • Tune the job. Fix skew, prefer broadcast joins for small dimensions, cache reused datasets and turn on Photon where the workload suits it.
Learn more

Full explanation

Attribution: make teams see their own spend

Attribution: make teams see their own spendyou can't optimize what no one owns

A single platform bill nobody can decompose is a bill nobody will reduce. Tag every cluster and job by team and pipeline, lean on Unity Catalog system tables for usage and surface a per-team breakdown. Showback (here's your spend) usually comes before chargeback (it hits your budget) - you earn the right to charge back by running clean, trusted showback first.