1 min lesson
The small-file problem
Explain your answer to "Why is the small-file problem so damaging at streaming scale and what are the two complementary fixes?" Add one concrete detail from the lesson.
Step 1 of 2
The small-file problemthe single most common scale failure
Streaming and micro-batch writes naturally produce many small files, because every micro-batch flushes a fresh Parquet file. Thousands of tiny files wreck read performance, because each one costs a metadata lookup and an open, and they inflate cost on object stores that charge per request.
- 1Compact on a schedule. Run Delta
OPTIMIZEto coalesce small files into right-sized ones of about 128MB-1GB, so downstream reads open a handful of files, not thousands. - 2Turn on auto-compaction / optimized writes. Let the writer target larger files during ingest so you generate fewer small files in the first place.
- 3Vacuum stale files. Remove tombstoned files past the retention window to reclaim storage, but keep enough history for time-travel replay.
The bronze write path and the silver/gold transforms must be independent jobs with their own compute. If a transform is slow or fails, ingest keeps landing raw events durably; if events spike, transforms drain the backlog at their own pace. Couple them and a single slow transform stalls the whole platform during exactly the traffic spike you built bronze to absorb.