Skip to lesson
Exit
Orchestration with Dagster1 / 2

1 min lesson

Backfills: replaying history safely

Rebuild the sequence in "Backfills: replaying history safely" from memory, ending with the check that proves the outcome.

Step 1 of 2

Backfills: replaying history safelythe move you make after a logic or schema change

A backfill re-materializes a range of historical partitions, typically after you fix a bug, change a transform or evolve a schema. The non-negotiable property is idempotency: re-running a partition must overwrite that slice cleanly, not append duplicates. Get this wrong and a backfill doubles your data.

  1. 1Scope the range. Identify exactly which partitions the change affects - a date window, a region, a set of customers - not the whole asset.
  2. 2Confirm idempotent writes. Each partition must overwrite-by-partition (Delta replaceWhere or a partition-scoped overwrite), never blind append.
  3. 3Stage and verify. Run a small slice first, diff row counts and key metrics against the old output before committing the full range.
  4. 4Throttle concurrency. Launch the backfill with capped parallelism so it does not starve the daily production runs sharing the cluster.
  5. 5Watch downstream. Let the asset graph re-materialize dependents of the backfilled slices and confirm gold metrics move as expected.