Lakehouse & Ingestion at Scale

A lakehouse is cheap object-store storage that behaves like a warehouse. Once you can say exactly which warehouse guarantees Delta Lake adds on top of plain files, the rest of this module is mechanics.

Classic architectures forced a choice. A data lake gave you cheap storage and open formats but no transactions, so concurrent writers corrupted tables and readers saw half-written data. A warehouse gave you ACID and schemas but locked your data in a proprietary engine and charged warehouse prices for every byte. The lakehouse keeps the lake's storage and adds the warehouse's guarantees through a metadata layer.

On Databricks that layer is Delta Lake. The data still lands as Parquet files in an object store like S3, but every table carries a transaction log - an ordered set of JSON commit files in a _delta_log directory that records which Parquet files belong to the table at each version.

Unity Catalog: the governance layerone place to grant, trace and audit

Tables alone don't make a platform governable. Unity Catalog sits above the workspaces and gives every table a three-level name - catalog.schema.table - with access control, lineage and audit attached to that name rather than to a cluster.

• Centralized RBACRole-Based Access Control. Granting permissions by role rather than configuring each person individually.. Grant on catalogs, schemas or tables once; it holds across every workspace and compute type instead of being re-pasted per cluster.
• Column and row controls. Column masks and row filters let you expose a table while hiding PIIPersonally Identifiable Information. Data that can identify a person (names, emails, SSNs); regulated and sensitive. columns or restricting rows by group - the lever you reach for with sensitive product data.
• Lineage and audit. Unity Catalog tracks table- and column-level lineage and logs access, which is how you answer "where did this number come from" and "who read this PIIPersonally Identifiable Information. Data that can identify a person (names, emails, SSNs); regulated and sensitive." without a forensic dig.

Why lakehouse over a warehouse hereone copy, three very different consumers

Cursor's data serves BI dashboards, product analytics and ML/agent training from the same tables. A warehouse-only stack would force you to copy data out to a separate object store every time the ML side needed raw Parquet, doubling storage and creating two sources of truth that drift. The lakehouse keeps a single governed copy that Spark, SQL and training jobs all read directly.

Medallion is one rule: each layer raises trust and shape and you never skip a layer to save a hop. Bronze preserves, silver cleans, gold serves.

The point of layering is that different consumers need different contracts. A replay or audit needs raw fidelity; a dashboard needs a clean conformed table; an agent feature store needs a narrow purpose-built shape. Collapsing those into one table means every consumer fights every other consumer's requirements.

What happens at each hopthe transforms that earn the layer

1. 1Land to bronze. Write incoming events append-only with idempotent keys; do not parse or reshape beyond what's needed to store them durably. This is the replay buffer for everything downstream.
2. 2Promote to silver. Deduplicate on event keys, enforce and cast the schema, drop or quarantine malformed rows and conform identifiers so a user_id means the same thing across sources.
3. 3Build gold. Aggregate into business marts or engineer agent/ML features from silver - narrow tables shaped for one query pattern, rebuildable from silver at any time.

The JD's "optimize the raw/bronze data layer for performance, reliability and cost at TB-to-PB scale" lives at the first hop. Bronze is where billions of events a day actually land, so its file layout and write path decide whether the rest of the platform is cheap or expensive to read.

Each layer gets its own contractSLAs, freshness and access differ by layer

The pattern follows the source, not the other way around. High-volume product events want a stream; SaaS systems of record want a connector. Pick wrong and you either melt a connector or build a streaming stack for a nightly Salesforce pull.

Cursor's two source families pull in opposite directions. First-party product telemetry is a firehose of editor and client events that arrives continuously. Third-party business data - billing, CRM, support - lives in SaaS systems you read on a schedule or via change capture.

Delivery semantics: the trade-off you must nameexactly-once is mostly idempotency in disguise

At billions of events a day, true end-to-end exactly-once is expensive and often unnecessary. The pragmatic pattern is at-least-once delivery plus idempotent writes, so duplicates are guaranteed to arrive but harmless because the write path collapses them.

Idempotency needs a key that's stable across retries - a producer-assigned event id or a deterministic hash of the payload plus timestamp. Then a MERGE into the Delta target or a dropDuplicates window in silver makes a replayed event a no-op rather than a double count.

Schema registry and evolutionkeep producers and the lakehouse from breaking each other

Hundreds of client builds emit events independently, so a producer will add a field or change a type without warning. A schema registry gives every event type a versioned schema and enforces compatibility rules at publish time, so a breaking change is rejected at the producer instead of corrupting bronze.

• Backward compatibility lets new consumers read old data - add optional fields, never repurpose an existing one.
• Forward compatibility lets old consumers tolerate new data - they ignore unknown fields rather than crash.
• Bronze stays permissive, silver stays strict. Land what arrives, then enforce the contract on the way to silver so a producer hiccup degrades one table, not the platform.

"Billions a day" is a number you should be able to convert to events per second in your head, then turn into partitions, file sizes and consumer counts. Interviewers watch for whether you do the arithmetic or wave at it.

Start with the math, out loud. A day is 86,400 seconds, so one billion events a day averages roughly 11,600 events/sec; five billion is about 58,000/sec. Traffic isn't flat, so apply a peak multiplier - developer usage clusters in working hours, so a 3-5x peak over the daily average is a reasonable planning assumption.

Partitioning without skewthe layout that enables pruning and parallelism

Partition bronze by event date (and maybe hour) so time-bounded queries and retention prune whole directories instead of scanning everything. Resist partitioning on a high-cardinality column like user_id directly - millions of tiny partitions is its own disaster. For clustering within a partition, use Z-ordering or Delta liquid clustering on the columns you actually filter on.

• Partition on low-to-medium cardinality time columns (date, hour) for pruning and retention.
• Watch for skew - a single hot tenant or a null key can pile most rows into one partition and one task; salt the key or use liquid clustering to spread the load.
• Cluster, don't over-partition - Z-order or liquid clustering on filter columns gives pruning without exploding the partition count.

The small-file problemthe single most common scale failure

Streaming and micro-batch writes naturally produce many small files - every micro-batch flushes a fresh Parquet file. Thousands of tiny files wreck read performance, because each one costs a metadata lookup and an open and they inflate cost on object stores that charge per request.

1. 1Compact on a schedule. Run Delta OPTIMIZE to coalesce small files into right-sized ones (~128MB-1GB), so downstream reads open a handful of files, not thousands.
2. 2Turn on auto-compaction / optimized writes. Let the writer target larger files during ingest so you generate fewer small files in the first place.
3. 3Vacuum stale files. Remove tombstoned files past the retention window to reclaim storage, but keep enough history for time-travel replay.

Most Spark slowness is one of three things: a shuffle you didn't need, skew you didn't handle or a join strategy you didn't pick. Name the bottleneck precisely and the fix is usually obvious.

A shuffle is Spark moving data across the network so rows that need to be together - same join key, same group - land on the same executor. It's the expensive operation: wide transforms like join, groupBy and distinct trigger it and a shuffle that spills to disk because it won't fit in memory is where jobs go to die.

Skew: the hot-key problemone tenant ruins the whole stage

Skew is when one key has far more rows than the others, so a single task processes most of the data while the rest sit idle. At Cursor a handful of huge enterprise tenants can dominate a groupBy user_id or org_id. The fixes are salting (append a random suffix to the hot key to split it across tasks, then re-aggregate) and Adaptive Query Execution, which detects skewed partitions at runtime and splits them automatically.

Broadcast vs shuffle joinsthe single highest-value join decision

-- AQE: re-optimizes the plan with runtime stats.
SET spark.sql.adaptive.enabled = true;
SET spark.sql.adaptive.skewJoin.enabled = true;        -- split skewed partitions
SET spark.sql.adaptive.coalescePartitions.enabled = true; -- shrink tiny shuffle partitions

-- Force a broadcast when you know the small side fits:
SELECT /*+ BROADCAST(dim) */ f.*, dim.name
FROM fact f JOIN dim ON f.dim_id = dim.id;

Caching and cluster sizing for costwhere the bill is set

Cache a DataFrame only when you reuse it multiple times in a job; caching something read once just wastes memory you needed for the shuffle. Photon is Databricks' vectorized C++ engine that speeds up scans and aggregations on the same cluster, so it's a near-free win for SQL-heavy ETL.

This is the design round in miniature: take Cursor's editor telemetry from the client to feature tables and narrate the trade-off at every hop. The structure below is one you can reuse cold.

Open by restating requirements before you draw anything - it's the move that separates a structured answer from a brain-dump.

The pipeline, hop by hopclient to feature tables

1. 1Client → ingest gateway. A thin gateway authenticates, validates the envelope, attaches a server-side receive timestamp and assigns or checks an event id for idempotency. It buffers and sheds load gracefully rather than failing the editor.
2. 2Gateway → stream (Kafka). Publish to topics partitioned for the peak throughput from section 4. The stream is the durable buffer that decouples bursty clients from downstream compute.
3. 3Stream → bronze Delta (idempotent). Spark Structured Streaming / Auto Loader appends raw events to bronze, partitioned by event date/hour, with a MERGE or dedup window on the event id so replays don't double-count.
4. 4Bronze → silver (dedup/conform). Deduplicate, enforce the registered schema, cast types, resolve late and out-of-order events with watermarks and conform identifiers. This is the trusted backbone.
5. 5Silver → gold / feature tables. Build the product-metric marts and the agent/ML feature tables, each shaped for its consumer and rebuildable from silver.

At each hop, call out the same four concerns the interviewer is listening for: partitioning, compaction, schema evolution and dead-letter handling. Saying "malformed events go to a dead-letter topic with the raw payload and error and I can replay them after a fix" is worth more than any extra box on the diagram.

SLAs and how you'd alert on eachfreshness and completeness are the two that matter

Failure modes and recoverythe part that proves you've run one of these