1 min lesson
Two ways to catch bad data
Match each case in "Two ways to catch bad data" to the signal and response that fit it.
Step 1 of 2
Two ways to catch bad datainline gates vs. out-of-band watchers
Run in the pipeline: Dagster asset checks, dbt tests, Great Expectations.
Deterministic assertions you already know to make: not_null, unique, accepted range, referential keys.
Can block: a failing check fails the asset and stops propagation.
Run beside the pipeline on a schedule, often statistical or learned.
Catches the unknown unknowns, like a slow null-rate creep no one wrote a rule for.
Usually alerts rather than blocks; noisier, needs a feedback loop to stay trusted.
Strong answers use both and say why. Inline tests encode the failures you can predict. Anomaly detection covers the ones you can't, but it earns trust slowly, so you start it in observe-only mode before letting it page anyone.
You own two failure classes. System reliability asks whether the job ran, finished and hit its window. Data quality asks whether the data the job produced is correct. A pipeline can be green, every task succeeded, and still write garbage because an upstream schema changed. Conflating the two is the classic junior tell.