Orchestration with Dagster

Orchestration is the layer that decides what runs, in what order, when and what to do when a step fails. On a lakehouse processing billions of events a day, it is the platform's nervous system.

Strip the buzzwords and an orchestrator does four things across the whole pipeline DAG: it sequences work so a transform never reads a half-written table, it schedules on a cadence or in response to events, it retries and recovers when a step dies and it observes every run so you know what is fresh, stale or broken. Without one you get a graveyard of cron jobs that nobody can reason about.

Dagster's distinctive bet is that you declare the data, not just the steps. In a task-centric tool like Airflow you write operators that run in an order you wired by hand; the framework has no idea those tasks produce a bronze_events table or that gold_daily_active depends on it. Dagster inverts this with software-defined assets: you declare the table or file each function produces and what it reads and the dependency graph is the data graph.

That fit with a lakehouse is why this matters for Cursor. Your assets span Databricks Spark jobs, dbt models and ingestion connectors and an asset graph gives you one control plane with lineage across all of them. You can stand in front of gold_retention and trace it back through silver to the raw Kafka topic without reading five repos.

Dagster vs the fieldbe ready to defend a choice, not recite features

An asset is a persisted data object plus the function that produces it and a declaration of what it reads. Once you think in assets, the pipeline stops being a pile of scripts and becomes a graph you can query.

The mechanics are small. You decorate a function with @asset, name its upstream dependencies as parameters and return the data. Dagster records that this function produces, say, silver_events and consumes bronze_events. Materializing an asset means running its function and persisting the result.

from dagster import asset

@asset
def bronze_events(context) -> None:
    # land raw product telemetry as-is into Delta (schema-on-read)
    ...

@asset(deps=[bronze_events])
def silver_events(context) -> None:
    # clean, dedupe, enforce schema; one row per real event
    ...

@asset(deps=[silver_events])
def gold_daily_active(context) -> None:
    # business aggregate read by BI and reverse-ETL
    ...

Because the edges are declared, the asset graph hands you three things you would otherwise build by hand: end-to-end lineage, selective materialization (rerun silver_events and everything downstream, nothing else) and a computed answer to “what is stale?” when an upstream changes.

Medallion layers map straight onto assetsthe dependency edges are the data flow

I/O managers: decouple compute from storagewhere data lands is config, not code

An I/O manager handles reading inputs and writing outputs so the asset function never hard-codes a path or a format. Your silver_events logic computes a DataFrame; a Delta I/O manager decides it gets written to a specific Delta table with the right options. Swap the manager and the same logic writes somewhere else, which keeps test, staging and prod clean.

• Compute logic stays pure: transform inputs, return outputs.
• Storage details (Delta path, partition columns, write mode) live in the I/O manager config.
• Different environments bind different managers - local Parquet in tests, governed Delta in prod.

Asset checks: quality at materialization timefail the run, not the dashboard

An asset check is a data-quality assertion attached to an asset and evaluated when it materializes: not-null on a key, row count within a band, no duplicate event ids. A failing check can block downstream materialization, so bad data stops at silver instead of surfacing as a wrong number in a gold dashboard three hops later.

from dagster import asset_check, AssetCheckResult

@asset_check(asset=silver_events)
def no_duplicate_event_ids(context) -> AssetCheckResult:
    dupes = count_duplicate_ids("silver_events")
    return AssetCheckResult(
        passed=dupes == 0,
        metadata={"duplicate_ids": dupes},
    )

At billions of events a day you never reprocess a whole table on a whim. You slice it into partitions and touch only the slices that changed.

A partition is a named slice of an asset. Dagster gives you three flavors and choosing the right one is most of the battle.

Partitions turn “process the table” into “materialize 2026-06-15, hour 14.” That single shift is what makes incremental materialization possible - each run computes one slice, so you never recompute billions of historical rows to add today's data.

Backfills: replaying history safelythe move you make after a logic or schema change

A backfill re-materializes a range of historical partitions, typically after you fix a bug, change a transform or evolve a schema. The non-negotiable property is idempotency: re-running a partition must overwrite that slice cleanly, not append duplicates. Get this wrong and a backfill doubles your data.

1. 1Scope the range. Identify exactly which partitions the change affects - a date window, a region, a set of customers - not the whole asset.
2. 2Confirm idempotent writes. Each partition must overwrite-by-partition (Delta replaceWhere or a partition-scoped overwrite), never blind append.
3. 3Stage and verify. Run a small slice first, diff row counts and key metrics against the old output before committing the full range.
4. 4Throttle concurrency. Launch the backfill with capped parallelism so it does not starve the daily production runs sharing the cluster.
5. 5Watch downstream. Let the asset graph re-materialize dependents of the backfilled slices and confirm gold metrics move as expected.

Partition mapping across grainshourly bronze feeding daily gold

Upstream and downstream assets often live at different grains. A partition mapping tells Dagster how a downstream partition depends on upstream ones - a single daily gold partition depends on the 24 hourly bronze partitions for that day. Declare the mapping and a backfill of one gold day pulls exactly the right 24 upstream slices, with no manual bookkeeping.

Something has to pull the trigger. Dagster gives you two: schedules for the clock, sensors for the world.

A schedule is the familiar case: run bronze_events at the top of every hour. A sensor is the event-driven one - it polls a source on an interval and launches a run when it sees a new file land, an upstream asset go fresh or an external system signal readiness. Sensors are how you stop polling-by-cron-and-hope and start reacting to actual data arrival.

Freshness as a declared SLAstop writing alerting glue by hand

A freshness policy (and the freshness checks that enforce it) lets you state an SLA on the asset itself: “gold_daily_active must be no more than 90 minutes behind its data.” Dagster evaluates it and flags the asset as overdue, so the SLA lives next to the definition instead of in a separate monitoring system that drifts out of sync.

Auto-materialization closes the loop: declare a policy and Dagster materializes an asset when its upstreams change or its freshness window is about to lapse, without a hand-written schedule for every node. You describe the desired state of the graph and let the system keep it true.

Protecting shared computeone backfill must not take down the cluster

• Retries with backoff so a transient Spark or network blip self-heals instead of paging someone.
• A run queue and concurrency limits so a flood of triggered runs does not all hit Databricks at once.
• Concurrency pools that cap how many runs touch a given resource (a cluster, a rate-limited API) in parallel.
• Idempotent assets, so a retried run is safe to replay (this is why the partition discipline from the last section pays off here).

The JD's example project is scaling Dagster for a growing population of technical users. That is a multi-tenant platform problem and it is the design they will hand you.

As the data team grows, one shared bag of definitions stops working. Teams step on each other's deploys, one team's backfill starves another's daily run and onboarding a new engineer means pulling you into the loop. The fix is structure on four axes.

Code locations isolate ownership and deploysthe multi-tenant primitive

A code location is an independently loaded set of Dagster definitions, usually one per team or domain. It gives each team its own deploy cycle and its own slice of the asset graph, while a single Dagster instance stitches them into one global view. A syntax error or a bad dependency in one location does not take down everyone else's pipelines.

The control-plane / compute splitthe single most important scaling decision

Run Dagster as the control plane and delegate heavy compute to Databricks and Spark. Dagster decides what to run, tracks lineage and freshness, enforces concurrency and submits work; the Spark cluster does the actual byte-crunching. If you try to execute billions of rows inside Dagster's own process you will fall over and you will also have coupled orchestration to compute so neither scales independently.

Self-serve is the multiplierturning recurring pain into a platform primitive

The role is judged on converting recurring data challenges into self-serve, scalable solutions. For orchestration that means scaffolding a new asset is a templated, guard-railed action: a generator that wires the I/O manager, requires an owner and a freshness policy and enforces a default partition scheme. The data team ships assets that are correct-by-construction and you are not the bottleneck on every new pipeline.