Skip to lesson
Exit
Capstone: Mock Loop & Self-Exam1 / 2

2 min lesson

Three Python drills

Name the key items in "Three Python drills", then explain why each one matters.

Step 1 of 2

Three Python drillsIdempotent transform, schema validation, backfill loop

  • Idempotent transform. A function that, given a partition path, writes deduped output and is safe to re-run - overwrite by partition, never append-then-hope.
  • Schema validation. Validate an incoming record against an expected contract; accept new nullable fields, reject a changed type, route the bad record to a dead-letter sink with a reason.
  • Backfill / partition loop. Iterate a date range, process one partition at a time, log progress and be resumable after a crash so you don't reprocess what already landed.
Resumable backfill loop - the detail interviewers probe is restartability
from datetime import date, timedelta

def backfill(start: date, end: date, sink) -> None:
    """Process one date partition at a time; skip what's done so a
    crash mid-run is safe to restart without double-processing."""
    day = start
    while day <= end:
        if sink.partition_complete(day):     # idempotency checkpoint
            day += timedelta(days=1)
            continue
        rows = transform_partition(day)       # pure, deterministic
        sink.overwrite_partition(day, rows)   # atomic per partition
        sink.mark_complete(day)
        print(f"backfilled {day}: {len(rows)} rows")
        day += timedelta(days=1)
Say it like this

“I'm processing per partition and checkpointing completion, so this is O(days) work but restartable - if it dies on day 40 of 90, the rerun skips the first 39. At billions of rows I'd also cap partition size and consider parallelizing across dates, watching for shuffle skew on the dedup step.”

Watch out

Practicing with the agent on is the trap this round exists to catch. Mute it, time each drill and aim for a clean, tested solution you can defend line by line. The discomfort of coding unaided is the signal that you'd been leaning on the tool, not a problem to prompt your way out of.