1 min lesson
Partitions, backfills and incremental processing
Pick two rows from the table in "Partitions, backfills and incremental processing" and explain the choice each one supports.
Step 1 of 2
At billions of events a day you never reprocess a whole table on a whim. You slice it into partitions and touch only the slices that changed.
A partition is a named slice of an asset. Dagster gives you three flavors and choosing the right one is most of the battle.
- Partition type
- Time-window
- Keyed by
- A date/hour bucket
- Example use
- Hourly bronze ingest, daily gold rollups
- Partition type
- Static
- Keyed by
- A fixed known set
- Example use
- Per-region or per-product-line tables
- Partition type
- Dynamic
- Keyed by
- A set discovered at runtime
- Example use
- Per-customer or per-new-source partitions
| Partition type | Keyed by | Example use |
|---|---|---|
| Time-window | A date/hour bucket | Hourly bronze ingest, daily gold rollups |
| Static | A fixed known set | Per-region or per-product-line tables |
| Dynamic | A set discovered at runtime | Per-customer or per-new-source partitions |
Time-window partitions carry most telemetry pipelines; static and dynamic cover the fan-out cases.
Partitions turn “process the table” into “materialize 2026-06-15, hour 14.” That single shift is what makes incremental materialization possible - each run computes one slice, so you never recompute billions of historical rows to add today's data.