2 min lesson
Partition mapping across grains
Respond to "Why is idempotency the critical property of a backfill and how do you achieve it on Delta?" Name the reason and the detail behind it.
Step 1 of 3
Partition mapping across grainshourly bronze feeding daily gold
Upstream and downstream assets often live at different grains. A partition mapping tells Dagster how a downstream partition depends on upstream ones - a single daily gold partition depends on the 24 hourly bronze partitions for that day. Declare the mapping and a backfill of one gold day pulls exactly the right 24 upstream slices, with no manual bookkeeping.
The expensive mistake is an accidental full-graph rerun. A non-partitioned asset wedged between partitioned ones or a check that touches the whole table, can force Dagster to recompute everything on every run. At PB scale that is a five-figure cluster bill from one careless edge. Keep the graph partitioned end to end and verify what a run actually selects before you launch it.
Cost control on this platform is mostly one discipline: never recompute a partition that did not change. Partitions plus partition mappings plus idempotent writes are the machinery that lets you enforce it. If you can explain how a logic change to one transform results in re-materializing only the affected slices and their dependents, you have demonstrated the cost instinct the role screens for.
When they probe cost on a big reprocess, model the scoped answer out loud: “A logic change to one transform doesn't mean I reprocess the table. I scope the backfill to the partitions that change, confirm the writes overwrite by partition so a re-run can't double the data, run one slice and diff it, then let the asset graph re-materialize only the affected dependents. That's the difference between a five-figure cluster bill and a bounded one.”
Learn more
Optional practice