1 min lesson
Two ways to catch bad data
Match each case in "Two ways to catch bad data" to the signal and response that fit it.
Step 1 of 2
Two ways to catch bad datainline gates vs. out-of-band watchers
Run in the pipeline: Dagster asset checks, dbt tests, Great Expectations.
Deterministic assertions you already know to make - not_null, unique, accepted range, referential keys.
Can block: a failing check fails the asset and stops propagation.
Run beside the pipeline on a schedule, often statistical or learned.
Catches the unknown unknowns - a slow null-rate creep no one wrote a rule for.
Usually alerts rather than blocks; noisier, needs a feedback loop to stay trusted.
Strong answers use both and say why. Inline tests encode the failures you can predict. Anomaly detection covers the ones you can't - but it earns trust slowly, so you start it in observe-only mode before letting it page anyone.
Two distinct failure classes and you own both. System reliability: did the job run, finish and hit its window? Data quality: is the data the job produced correct? A pipeline can be green - every task succeeded - and still write garbage because an upstream schema changed. Conflating the two is the classic junior tell.