Skip to lesson
Exit
Capstone: Mock Loop & Self-Exam1 / 2

2 min lesson

A 4-hour timebox that ends with something demoable

Take "A 4-hour timebox that ends with something demoable" step by step, then finish with the result that proves it worked.

Step 1 of 2

A 4-hour timebox that ends with something demoableSlice, don't sprawl

  1. 10:00–0:20 · Scope and freeze. Write the smallest end-to-end slice on paper: one event type, bronze → silver → gold, one quality check. Cut everything else to the README's what I'd do next.
  2. 20:20–1:30 · Bronze + silver. Land raw, then dedup with an idempotent MERGE. Get a re-run to produce identical row counts - that's your idempotency proof.
  3. 31:30–2:30 · Gold + quality check. Build one aggregate and one assertion (row count, null rate or freshness). Make the assertion fail loudly on bad input.
  4. 42:30–3:20 · Tests and a re-run. Unit-test the transform, then run the whole pipeline twice and diff the output to prove idempotency.
  5. 53:20–4:00 · README. Trade-offs you made, what you cut and why and the next three things you'd build. This is graded as hard as the code.
The idempotent heart of the build - an upsert keyed on event id, defensible line by line
-- silver.events: dedup bronze to the latest record per event_id.
-- Re-running this MERGE on the same bronze partition is a no-op:
-- that property is the whole point of the exercise.
MERGE INTO silver.events AS t
USING (
  SELECT *
  FROM (
    SELECT *,
      ROW_NUMBER() OVER (
        PARTITION BY event_id
        ORDER BY ingested_at DESC, _commit_version DESC
      ) AS rn
    FROM bronze.events
    WHERE event_date = :run_date   -- incremental: one partition only
  )
  WHERE rn = 1
) AS s
ON t.event_id = s.event_id
WHEN MATCHED AND s.ingested_at > t.ingested_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Be ready to defend every clause. Why ROW_NUMBER over event_id and not DISTINCT. Why the event_date filter makes this incremental. Why the MATCHED AND guard protects against an out-of-order late record overwriting a newer one. If you can't justify a line, an interviewer will find it.

Learn more

Full explanation

Prove it, don't claim it

Prove it, don't claim it

Idempotency is verifiable, so verify it in the demo: run the pipeline, capture the gold row count, run it again, show the count is unchanged. A 20-second live diff beats a paragraph in the README asserting the pipeline is idempotent.

Watch out

The classic failure is spending three hours on an elegant config-driven framework that ingests nothing yet. The onsite measures product sense and autonomy - a working bronze-to-gold slice with one real quality check and an honest README outscores a beautiful abstraction with no output. Timebox ruthlessly and demo something that runs.