2 min lesson
Caching and cluster sizing for cost
Name the parts in "Caching and cluster sizing for cost" and give the practical job of each one.
Step 1 of 2
Caching and cluster sizing for costwhere the bill is set
Cache a DataFrame only when you reuse it multiple times in a job; caching something read once just wastes memory you needed for the shuffle. Photon is Databricks' vectorized C++ engine that speeds up scans and aggregations on the same cluster, so it's a near-free win for SQL-heavy ETL.
- Cores
- Match parallelism to shuffle-partition count; too few cores serializes tasks
- Memory
- Size to avoid shuffle spill on your largest stage, not your smallest
- Autoscaling
- Scale workers with the job's stages; cap the max so a runaway job can't run up the bill
- Spot vs on-demand
- Spot for fault-tolerant batch (huge savings, can be preempted); on-demand for SLA-bound or driver nodes
When a Spark job is slow in the interview, don't jump to "add more workers." Ask for the Spark UI signal first: "Is one task much slower than the rest?" (skew) or "How big is the shuffle read?" (a wide transform to push down or a join to broadcast). Diagnosing from the stage view before touching cluster size is exactly the seniority signal - throwing hardware at skew just buys idle executors.
Spot instances are a real cost win on fault-tolerant batch, but never put the driver or an SLA-critical streaming job on pure spot - a preemption kills the driver and the whole job dies. The clean answer is spot for stateless batch workers, on-demand for the driver and anything with a freshness SLA.