2 min lesson
The control-plane / compute split
Choose the best answer to "When scaling Dagster for many teams, why is it essential to keep Dagster as a control plane and delegate compute to Databricks/Spark?" Then say what led you there.
Step 1 of 2
The control-plane / compute splitthe single most important scaling decision
Run Dagster as the control plane and delegate heavy compute to Databricks and Spark. Dagster decides what to run, tracks lineage and freshness, enforces concurrency and submits work; the Spark cluster does the actual byte-crunching. If you try to execute billions of rows inside Dagster's own process you will fall over and you will also have coupled orchestration to compute so neither scales independently.
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
Top to bottom: what users touch, what the control plane owns and where the bytes actually move.
Learn more
Optional practice