2 min lesson
Caching and cluster sizing for cost
Name the parts in "Caching and cluster sizing for cost" and give the practical job of each one.
Step 1 of 2
Caching and cluster sizing for costwhere the bill is set
Cache a DataFrame only when you reuse it multiple times in a job; caching something read once just wastes memory you needed for the shuffle. Photon is Databricks' vectorized C++ engine that speeds up scans and aggregations on the same cluster, so it's a near-free win for SQL-heavy ETL.
- Cores
- Match parallelism to shuffle-partition count; too few cores serializes tasks
- Memory
- Size to avoid shuffle spill on your largest stage, not your smallest
- Autoscaling
- Scale workers with the job's stages; cap the max so a runaway job can't run up the bill
- Spot vs on-demand
- Spot for fault-tolerant batch, which saves a lot but can be preempted; on-demand for SLA-bound or driver nodes
When a Spark job is slow in the interview, don't jump to "add more workers." Ask for the Spark UI signal first. "Is one task much slower than the rest?" points to skew. "How big is the shuffle read?" points to a wide transform to push down or a join to broadcast. Diagnosing from the stage view before touching cluster size is the seniority signal, because throwing hardware at skew just buys idle executors.
Spot instances are a real cost win on fault-tolerant batch, but never put the driver or an SLA-critical streaming job on pure spot, because a preemption kills the driver and the whole job dies. The clean answer is spot for stateless batch workers, on-demand for the driver and anything with a freshness SLA.