2 min lesson
Lakehouse fundamentals on Databricks
Answer "What does the Delta transaction log provide that a directory of plain Parquet files cannot and why does it matter at Cursor's scale?" Use the clue you found.
Step 1 of 2
A lakehouse is cheap object-store storage that behaves like a warehouse. Once you can say exactly which warehouse guarantees Delta Lake adds on top of plain files, the rest of this module is mechanics.
Classic architectures forced a choice. A data lake gave you cheap storage and open formats but no transactions, so concurrent writers corrupted tables and readers saw half-written data. A warehouse gave you ACID and schemas but locked your data in a proprietary engine and charged warehouse prices for every byte. The lakehouse keeps the lake's storage and adds the warehouse's guarantees through a metadata layer.
On Databricks that layer is Delta Lake. The data still lands as Parquet files in an object store like S3, but every table carries a transaction log - an ordered set of JSON commit files in a _delta_log directory that records which Parquet files belong to the table at each version.
- ACID commits
- Each write is an atomic log entry; readers see a complete version or the previous one, never a partial table
- Time travel
- Query any past version (
VERSION AS OF) because the log knows the file set at every commit - replay and audit for free - Schema enforcement
- Writes that don't match the table schema are rejected at commit, so bad data never lands silently
- Schema evolution
- Add columns or widen types deliberately (
mergeSchema) without rewriting history - Concurrency
- Optimistic concurrency on the log lets many writers commit safely instead of corrupting files
The log is the table. The Parquet files are just storage it points at.