1 min lesson
Spark performance internals
Put this idea into your own words: "Most Spark slowness is one of three things."
Step 1 of 3
Most Spark slowness is one of three things: a shuffle you didn't need, skew you didn't handle or a join strategy you didn't pick. Name the bottleneck precisely and the fix is usually obvious.
A shuffle is Spark moving data across the network so rows that need to be together - same join key, same group - land on the same executor. It's the expensive operation: wide transforms like join, groupBy and distinct trigger it and a shuffle that spills to disk because it won't fit in memory is where jobs go to die.
Learn more
Advanced table
Match the symptom to the cause out loud
- Symptom
- A few tasks run forever, rest finish fast
- Likely cause
- Data skew (hot key / null key)
- Lever
- Salt the key, AQE skew join, skew hint
- Symptom
- Huge shuffle read/write, disk spill
- Likely cause
- Unnecessary or oversized wide transform
- Lever
- Filter/aggregate earlier, broadcast the small side, raise shuffle partitions
- Symptom
- Join is slow, one side is small
- Likely cause
- Shuffle join where a broadcast would do
- Lever
- Broadcast join (auto via AQE or hint)
- Symptom
- Repeated scans of the same DataFrame
- Likely cause
- Recomputing instead of reusing
- Lever
- Cache/persist the reused dataset
- Symptom
- Job is just CPU-bound on scans
- Likely cause
- No vectorization
- Lever
- Enable Photon for vectorized execution
| Symptom | Likely cause | Lever |
|---|---|---|
| A few tasks run forever, rest finish fast | Data skew (hot key / null key) | Salt the key, AQE skew join, skew hint |
| Huge shuffle read/write, disk spill | Unnecessary or oversized wide transform | Filter/aggregate earlier, broadcast the small side, raise shuffle partitions |
| Join is slow, one side is small | Shuffle join where a broadcast would do | Broadcast join (auto via AQE or hint) |
| Repeated scans of the same DataFrame | Recomputing instead of reusing | Cache/persist the reused dataset |
| Job is just CPU-bound on scans | No vectorization | Enable Photon for vectorized execution |
Match the symptom to the cause out loud - that's what the Spark probe is actually testing.