1 min lesson
Cost optimization at TB/PB scale
Explain the practical point behind "The JD's recurring trio is scale, reliability, and cost."
Step 1 of 4
The JD's recurring trio is scale, reliability, and cost. At billions of events a day, a sloppy storage layout or an always-on cluster isn't a rounding error - it's a line item someone in finance will ask you about.
Cost optimization splits cleanly into two bills: what you pay to store the data and what you pay to compute over it. Treat them separately, because the levers are different and a good answer touches both.
Learn more
Advanced table
Reference table
- Lever
- Compaction
- Problem it fixes
- Millions of tiny files from streaming writes - slow reads, metadata bloat
- What you actually do
- Periodic OPTIMIZE / compaction to target ~128MB–1GB files
- Lever
- Partitioning
- Problem it fixes
- Every query scans the whole table
- What you actually do
- Partition by a high-selectivity column (usually date/hour) so the engine prunes
- Lever
- Z-ordering / clustering
- Problem it fixes
- Even within a partition, related rows are scattered
- What you actually do
- Cluster on common filter columns so scans skip irrelevant files
- Lever
- Lifecycle / retention
- Problem it fixes
- Raw data accumulating forever in expensive hot storage
- What you actually do
- Expire or tier past the replay window; vacuum old Delta versions
| Lever | Problem it fixes | What you actually do |
|---|---|---|
| Compaction | Millions of tiny files from streaming writes - slow reads, metadata bloat | Periodic OPTIMIZE / compaction to target ~128MB–1GB files |
| Partitioning | Every query scans the whole table | Partition by a high-selectivity column (usually date/hour) so the engine prunes |
| Z-ordering / clustering | Even within a partition, related rows are scattered | Cluster on common filter columns so scans skip irrelevant files |
| Lifecycle / retention | Raw data accumulating forever in expensive hot storage | Expire or tier past the replay window; vacuum old Delta versions |
The small-file problem is the one almost every fast-growing platform hits - streaming ingestion writes constantly and nobody compacts until reads slow to a crawl.
A good partition scheme means a query for "yesterday" reads one day of files, not the whole table. You pay less for the bytes scanned (compute) and the engine touches less of the object store (I/O). Over-partition, though - say by user_id - and you recreate the small-file problem with one tiny file per partition. The art is picking a column selective enough to prune but coarse enough to keep files fat.