2 min lesson
A 4-hour timebox that ends with something demoable
Take "A 4-hour timebox that ends with something demoable" step by step, then finish with the result that proves it worked.
Step 1 of 2
A 4-hour timebox that ends with something demoableSlice, don't sprawl
- 10:00–0:20 · Scope and freeze. Write the smallest end-to-end slice on paper: one event type, bronze → silver → gold, one quality check. Cut everything else to the README's what I'd do next.
- 20:20–1:30 · Bronze + silver. Land raw, then dedup with an idempotent MERGE. Get a re-run to produce identical row counts - that's your idempotency proof.
- 31:30–2:30 · Gold + quality check. Build one aggregate and one assertion (row count, null rate or freshness). Make the assertion fail loudly on bad input.
- 42:30–3:20 · Tests and a re-run. Unit-test the transform, then run the whole pipeline twice and diff the output to prove idempotency.
- 53:20–4:00 · README. Trade-offs you made, what you cut and why and the next three things you'd build. This is graded as hard as the code.
-- silver.events: dedup bronze to the latest record per event_id.
-- Re-running this MERGE on the same bronze partition is a no-op:
-- that property is the whole point of the exercise.
MERGE INTO silver.events AS t
USING (
SELECT *
FROM (
SELECT *,
ROW_NUMBER() OVER (
PARTITION BY event_id
ORDER BY ingested_at DESC, _commit_version DESC
) AS rn
FROM bronze.events
WHERE event_date = :run_date -- incremental: one partition only
)
WHERE rn = 1
) AS s
ON t.event_id = s.event_id
WHEN MATCHED AND s.ingested_at > t.ingested_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;Be ready to defend every clause. Why ROW_NUMBER over event_id and not DISTINCT. Why the event_date filter makes this incremental. Why the MATCHED AND guard protects against an out-of-order late record overwriting a newer one. If you can't justify a line, an interviewer will find it.
Learn more
Full explanation
Prove it, don't claim it
Idempotency is verifiable, so verify it in the demo: run the pipeline, capture the gold row count, run it again, show the count is unchanged. A 20-second live diff beats a paragraph in the README asserting the pipeline is idempotent.
The classic failure is spending three hours on an elegant config-driven framework that ingests nothing yet. The onsite measures product sense and autonomy - a working bronze-to-gold slice with one real quality check and an honest README outscores a beautiful abstraction with no output. Timebox ruthlessly and demo something that runs.