2 min lesson

Skew: the hot-key problem

Put yourself in this case. "A daily aggregation by org_id has stalled: 198 of 200 tasks finished in two minutes, two have run for forty. What's happening and how do you fix it?" Give the clearest next step.

Skew: the hot-key problemone tenant ruins the whole stage

Skew is when one key has far more rows than the others, so a single task processes most of the data while the rest sit idle. Picture a handful of huge tenants dominating a groupBy user_id or org_id. The fixes are salting, which appends a random suffix to the hot key to split it across tasks and then re-aggregates, and Adaptive Query Execution, which detects skewed partitions at runtime and splits them automatically.

Learn more

Full explanation

Broadcast vs shuffle joins

Broadcast vs shuffle joinsthe single highest-value join decision

Broadcast join

Small side fits in executor memory

Ship the small table to every executor, so the big one never shuffles

AQE auto-picks it under the broadcast threshold

Sort-merge shuffle join

Both sides large; neither broadcasts

Shuffle both by key so matches co-locate

Correct but costly, so minimize the data shuffled

Adaptive Query Execution handles skew and join strategy at runtime. Keep it on.

-- AQE: re-optimizes the plan with runtime stats.
SET spark.sql.adaptive.enabled = true;
SET spark.sql.adaptive.skewJoin.enabled = true;        -- split skewed partitions
SET spark.sql.adaptive.coalescePartitions.enabled = true; -- shrink tiny shuffle partitions

-- Force a broadcast when you know the small side fits:
SELECT /*+ BROADCAST(dim) */ f.*, dim.name
FROM fact f JOIN dim ON f.dim_id = dim.id;

Learn more

Optional practice

Skew: the hot-key problem

Step 1 of 3