1 min lesson
Spark performance internals
Put this idea into your own words. "Most Spark slowness is one of three things."
Step 1 of 3
Most Spark slowness is one of three things: a shuffle you didn't need, skew you didn't handle or a join strategy you didn't pick. Name the bottleneck precisely and the fix is usually obvious.
A shuffle is Spark moving data across the network so rows that need to be together, such as the same join key or the same group, land on the same executor. It's the expensive operation: wide transforms like join, groupBy and distinct trigger it and a shuffle that spills to disk because it won't fit in memory is where jobs go to die.
Learn more
Advanced table
Match the symptom to the cause out loud
- Symptom
- A few tasks run forever, rest finish fast
- Likely cause
- Data skew from a hot key or null key
- Lever
- Salt the key, AQE skew join, skew hint
- Symptom
- Huge shuffle read/write, disk spill
- Likely cause
- Unnecessary or oversized wide transform
- Lever
- Filter/aggregate earlier, broadcast the small side, raise shuffle partitions
- Symptom
- Join is slow, one side is small
- Likely cause
- Shuffle join where a broadcast would do
- Lever
- Broadcast join, automatic via AQE or by hint
- Symptom
- Repeated scans of the same DataFrame
- Likely cause
- Recomputing instead of reusing
- Lever
- Cache/persist the reused dataset
- Symptom
- Job is just CPU-bound on scans
- Likely cause
- No vectorization
- Lever
- Enable Photon for vectorized execution
| Symptom | Likely cause | Lever |
|---|---|---|
| A few tasks run forever, rest finish fast | Data skew from a hot key or null key | Salt the key, AQE skew join, skew hint |
| Huge shuffle read/write, disk spill | Unnecessary or oversized wide transform | Filter/aggregate earlier, broadcast the small side, raise shuffle partitions |
| Join is slow, one side is small | Shuffle join where a broadcast would do | Broadcast join, automatic via AQE or by hint |
| Repeated scans of the same DataFrame | Recomputing instead of reusing | Cache/persist the reused dataset |
| Job is just CPU-bound on scans | No vectorization | Enable Photon for vectorized execution |
Match the symptom to the cause out loud. That's what the Spark probe is actually testing.