Skip to lesson
Exit
Lakehouse & Ingestion at Scale1 / 2

1 min lesson

What happens at each hop

Take this case and make the call: "Which layer carries the JD's mandate to 'optimize the raw/bronze layer for performance, reliability and cost at TB-to-PB scale,' and why is its file layout so consequential?"

Step 1 of 2

What happens at each hopthe transforms that earn the layer

  1. 1Land to bronze. Write incoming events append-only with idempotent keys; do not parse or reshape beyond what's needed to store them durably. This is the replay buffer for everything downstream.
  2. 2Promote to silver. Deduplicate on event keys, enforce and cast the schema, drop or quarantine malformed rows and conform identifiers so a user_id means the same thing across sources.
  3. 3Build gold. Aggregate into business marts or engineer agent/ML features from silver - narrow tables shaped for one query pattern, rebuildable from silver at any time.

The JD's "optimize the raw/bronze data layer for performance, reliability and cost at TB-to-PB scale" lives at the first hop. Bronze is where billions of events a day actually land, so its file layout and write path decide whether the rest of the platform is cheap or expensive to read.