1 min lesson
Protecting shared compute
Answer "When would you reach for a sensor instead of a schedule and what is the operational gotcha with sensors?" Then point to the lesson detail behind your choice.
Step 1 of 2
Protecting shared computeone backfill must not take down the cluster
- Retries with backoff so a transient Spark or network blip self-heals instead of paging someone.
- A run queue and concurrency limits so a flood of triggered runs does not all hit Databricks at once.
- Concurrency pools that cap how many runs touch a given resource (a cluster, a rate-limited API) in parallel.
- Idempotent assets, so a retried run is safe to replay (this is why the partition discipline from the last section pays off here).
“I replace fragile cron jobs with an observable, dependency-aware control plane. Schedules cover predictable cadence, sensors react to data actually arriving and freshness policies make the SLA a declared property of the asset. When something runs late, I see which asset is overdue and why, instead of discovering it from a stakeholder asking why the dashboard is stale.”
Sensors poll, so a chatty sensor on a tight interval against a slow source becomes its own load problem and can miss or double-fire if it is not cursor-based. Track a cursor so each tick only considers new work and tune the interval to the source - not every sensor should run every five seconds.