2 min lesson
The small-file problem
Explain your answer to "Why is the small-file problem so damaging at streaming scale and what are the two complementary fixes?" Add one concrete detail from the lesson.
Step 1 of 2
The small-file problemthe single most common scale failure
Streaming and micro-batch writes naturally produce many small files - every micro-batch flushes a fresh Parquet file. Thousands of tiny files wreck read performance, because each one costs a metadata lookup and an open and they inflate cost on object stores that charge per request.
- 1Compact on a schedule. Run Delta
OPTIMIZEto coalesce small files into right-sized ones (~128MB-1GB), so downstream reads open a handful of files, not thousands. - 2Turn on auto-compaction / optimized writes. Let the writer target larger files during ingest so you generate fewer small files in the first place.
- 3Vacuum stale files. Remove tombstoned files past the retention window to reclaim storage, but keep enough history for time-travel replay.
The bronze write path and the silver/gold transforms must be independent jobs with their own compute. If a transform is slow or fails, ingest keeps landing raw events durably; if events spike, transforms drain the backlog at their own pace. Couple them and a single slow transform stalls the whole platform during exactly the traffic spike you built bronze to absorb.
Don't quote "billions a day" and then propose a layout sized for the daily average. The number that breaks you is the working-hours peak, which can be several times the average. Stating a peak multiplier and sizing partitions and consumers against it is the difference between a design that holds and one that pages you at 9am.