11 · Lake, Warehouse, Lakehouse

On Friday morning, a message from finance was waiting for Theo. He read it aloud.

“Chargeback overnight on an August order. Order 3993289, $17.24. Status is now refunded.”

A chargeback happens when a customer’s bank takes a payment back from a shop, usually because the customer disputed the charge. For Steep, it works like a refund that arrives late.

“August changes every day,” said Theo. “Since your first Monday, not one day has passed without a refund for an August order.”

Mia looked up from her notebook. “August is over.”

“The month is over. Its orders are not.” Theo pointed at the message. “This one was placed on 19 August. The bank took 37 days to take the money back.”

He opened the orders table on his screen. Steep’s warehouse tables are plain Hive tables on the lake: one folder per day and city, as in Chapter 9. “Every night, our job copies changes like this into the lake. But the files in a folder are written once. Nobody edits them. So to change one row, the job rewrites the whole folder of the day when the order was placed. Every row in it.”

Mia was thinking about something else. On her first Monday, Dana had asked her question. Since then, Mia had counted orders many times. Each time, she had counted from a table that kept changing.

“Dana asked on 14 September,” she said. “Today’s August is not the August of that Monday. What did the warehouse say on 14 September?”

“It cannot tell you,” said Theo. “When the job rewrites a folder, it deletes the old files. A Hive table has no memory. It only knows today.”

“But Steep keeps the database’s diary,” said Mia. “The saved copy of the binlog.”

Theo looked at her for a moment. Then he smiled. “Then let us build a time machine.”

Mia wrote at the top of a new page: What did we know, and when did we know it?

ImportantThe big idea

Data that changes needs a table that remembers its changes.

A calm lake at sunset, its surface scattered with floating leaves in navy, tomato and mustard. On the right, a red wooden boathouse stands on posts over the water at the end of a wooden dock. Its open front shows three shelves of identical blank jars, six on each shelf, under a hanging lamp. Pine trees and rocks line the shore.

Look at the picture: a lake full of floating leaves, and a tidy boathouse with shelves of identical jars. The lake holds whatever falls into it. The boathouse holds only what someone has put on a shelf. And the boathouse does not stand on the shore. It stands on posts, in the lake. By the end of this chapter, you will see why.

The lake and the warehouse

In Chapters 9 and 10, you met Steep’s data lake: files kept in cheap storage, such as HDFS or object storage, in whatever format they arrived. A lake holds anything: tables in Parquet, events in JSON, logs, even images. Many engines can read the same files. Each reader applies a schema when it reads (schema-on-read). In the picture, the lake is the water, and the files are the leaves: many shapes, floating where they fell.

A data warehouse is the older idea. It is a database built for analysis. It stores its tables in its own managed format, checks every row against the schema when the row is written (schema-on-write), and lets many people query the same tables at once with SQL. It also controls who may read what. Companies such as Teradata have built warehouse systems for decades; today, cloud warehouses such as Snowflake, Google BigQuery and Amazon Redshift do the same job. In the picture, the warehouse is the boathouse: every jar on a shelf.

Data lake Data warehouse
Stores Files, in any format Tables, in the warehouse’s own format
Schema Applied when you read Checked when you write
Cost of storage Low Higher
Changing one row Hard (rewrite files) Easy (UPDATE)
Who can read it Many engines Mostly the warehouse’s own engine
Main risk Nobody knows what the files mean Cost, and your data is locked inside

A word about names. Until now, this book has used “the warehouse” the way Steep’s staff do: for the whole data platform, the place where Steep collects and cleans its data (the Prologue’s definition). Strictly, Steep keeps that data as files in a lake: Parquet files in folders, described by Hive’s metastore (Chapters 9 and 10). On top of those files, it builds warehouse-style tables in layers, from raw copies to clean summaries (Chapter 12). So Steep’s “warehouse” is a set of plain Hive tables on a lake. A warehouse that stands on a lake has its own name, the lakehouse, and this chapter ends there.

A lake where nobody knows what the files mean, or which ones are current, is often called a data swamp. Steep’s lake was not a swamp. But it had a quieter weakness, and the chargeback found it.

Why a Hive table cannot take back a refund

Remember from Chapter 10: a file in HDFS is written once. Object storage works the same way: you can replace a whole file, but you cannot change a few bytes inside it. Parquet files are built to be written once too. So a plain Hive table cannot update a row (change it in place).

To record the chargeback, Steep’s nightly job must rewrite a whole partition: the folder dt=2026-08-19/city=harbor. In Hive’s SQL, that is INSERT OVERWRITE … PARTITION (…), which replaces everything in the folder. The job reads the whole folder, changes one row, and writes it all back: 2,533 rows, for one refund.

That one chargeback was not alone. From 15 September to the end of 25 September, 200 August orders were refunded, on every one of those eleven days. August has 124 folders in the lake (1 August to 31 August, four cities each). Each night, the job rewrites every folder that holds one of that day’s changes. Over those eleven nights, recording 200 changed rows means 171 folder rewrites and 324,247 rows written: about 1,621 rows written for each row that changed.

Rewriting is only the first problem. There are four more.

  • No safe moment to read. While the job replaces a folder’s files, a query that reads the folder can see some old files and some new ones, or none at all. Its answer can be wrong without any error.
  • No safe way to write twice. If two jobs rewrite the same folder at the same time, the last one wins, and the other one’s change is lost.
  • No memory. When a folder is overwritten, the old files are deleted. Nobody can ask what the table said last week. This is Mia’s question.
  • Slow planning. Hive’s metastore knows the table’s partitions, but not its files. To plan a query, an engine lists the files in every folder it needs. On object storage, listing thousands of folders takes time, and the small-files problem of Chapter 9 makes it worse.

Hive itself later added transactional tables that can update and delete rows. But they work only with ORC files, and other engines support them only in part. Most lakes kept their plain tables, with these problems.

A time machine made of the binlog

Mia did not need a new table to answer her question. MySQL had deleted most of its own August binlog files by now (after 30 days, as Chapter 7 explained). But she had Steep’s saved copy of the binlog, the lake table cdc_orders_binlog: every change to every order, in commit order. To see the orders table as it stood at the end of 14 September, replay every change up to that moment and stop. Then count August.

Table 1: August’s orders (placed 1–31 August), replayed from the binlog as of the end of three days.
As of Completed orders Net revenue Refunded orders
End of 14 September (Dana’s question) 194,276 $2,419,745.03 1,679
End of 25 September (this chapter) 194,076 $2,417,213.40 1,879
End of the data, 25 October 193,976 $2,415,954.51 1,979

At the end of the day Dana asked her question, the database said August had 194,276 completed orders and $2,419,745.03 of net revenue. By the end of that Friday, 200 of those orders had been refunded, and August had lost $2,531.63. (The table counts whole days. At three o’clock that Friday, when Mia counted, 192 of these refunds had arrived; the other eight came that evening.)

This book’s data runs to 25 October, so we can look further ahead than Mia could. (The headline numbers in this book use final statuses; see Chapter 1.) The last August order changed on 10 October. In total, 300 August orders and $3,790.52 left August after 14 September. Each of these refunds arrived between 16 and 40 days after its order, 33 days in the middle. Of all of August’s 1,979 refunds up to 25 October, 1,237 had arrived by the time the month ended.

Show the code 
after = asof[asof.as_of_date >= CLOSE_DAY]
fig, ax = bk.figure(8, 3.9)
ax.step(after.as_of_date, after.completed_orders, where="post", color=bk.INK, linewidth=2)
marks = [(FIRST_REPORT, first, bk.TEAL, "14 Sep: Dana's question", (8, 10)),
         (STORY_DAY, story, bk.TOMATO, f"{STORY_DAY.day} Sep: this chapter", (8, 10)),
         (asof.as_of_date.iloc[-1], final, bk.INK, "final", (-30, 10))]
for day, t, color, label, offset in marks:
    ax.plot([day], [t["completed_orders"]], "o", color=color, markersize=7, zorder=3)
    ax.annotate(f"{label}\n{bk.fmt_int(t['completed_orders'])}", (day, t["completed_orders"]),
                xytext=offset, textcoords="offset points", fontsize=9, color=bk.INK)
ax.set_ylim(final["completed_orders"] - 120, at_close.completed_orders + 120)
ax.yaxis.set_major_formatter(lambda v, _: f"{v:,.0f}")
ax.xaxis.set_major_locator(mdates.WeekdayLocator(byweekday=mdates.MO, interval=2))
ax.xaxis.set_major_formatter(lambda v, _: f"{mdates.num2date(v).day} {mdates.num2date(v):%b}")
ax.set_ylabel("Completed August orders")
ax.set_title(f"From 14 September to {day_month(world.END)}, late refunds took {bk.fmt_int(n_all)} orders out of August")
plt.show()
A line that starts high on 31 August and steps down almost every day until early October, then stays flat until 25 October. A teal dot marks 14 September, the day of Dana's question, and a tomato dot marks 25 September, the day of this chapter.
Figure 1: August’s completed orders, as the database showed them at the end of each day after the month ended. The axis does not start at zero.

Replaying a binlog works, but it is slow and clumsy. To ask about one moment, you replay the history of every order. Theo had a better idea for the lake.

A table that keeps a diary

The idea behind modern lake tables fits on one napkin, and Theo drew it. On the left, a folder of Parquet files. On the right, a small notebook.

“Keep the files,” he said. “Never change them. But keep a list.”

Data files never change. New or changed rows go into new files. Old files stay as they are.

A list says which files make up the table. Next to the data files sits metadata: data about data. Its main job is to say which files form the table right now. Each version of that list is a snapshot: the table as it was after one change.

A change is a commit. To change the table, a writer first writes its new data files. Then it writes a new snapshot that lists the new set of files. Last, it makes the new snapshot the current one, in a single step. This last step is the commit. (Commit means “make final”. In Chapter 7, the database commits a piece of work; in Chapter 8, a reader commits its bookmark. Here, it is the moment a change becomes part of the table.) In Apache Iceberg, the commit changes one pointer: a small record that says which metadata file is the current one. A catalog, a small service, keeps these pointers for every table. In Delta Lake, the commit writes the next numbered file in the table’s log. If another writer committed first, the commit fails. The writer then checks the newer snapshot and tries again, or gives up if the two changes clash.

A reader starts from the current snapshot and reads exactly the files it lists. A change that is half written is not in any snapshot yet, so no reader can see it. This gives the table four promises, known by the letters ACID:

  • Atomic: a change happens completely, or not at all.
  • Consistent: every commit takes the table from one valid state to another.
  • Isolated: readers and writers working at the same time do not see each other’s half-done work.
  • Durable: once a change is committed, it stays.

The snapshots also give three gifts.

Time travel. Old snapshots stay until someone deletes them. So you can ask for the table as it was at an earlier moment. In Spark SQL, on an Iceberg or Delta Lake table:

SELECT count(*) AS completed_orders, sum(net_amount) AS net_revenue
FROM ods_orders TIMESTAMP AS OF '2026-09-14 23:59:59'
WHERE dt BETWEEN '2026-08-01' AND '2026-08-31'
  AND status = 'completed';

Is that Mia’s question in one query? Only if the table was kept up to date. If CDC changes were merged into ods_orders every few minutes, this query would return almost the same answer as the replay. But Steep loads once a night, at 2 a.m. (Chapter 13). The last load before 23:59 on 14 September held changes up to the end of 13 September. One more detail: the engine reads the time in the query in the time zone of your connection, so set it to Steep’s.

Schema evolution. A schema can change over time: a new column, a renamed column, a wider number type. The metadata records each change, so old files need not be rewritten. Iceberg gives every column a permanent ID, so renaming a column never mixes up its data with another column’s.

Merge. A table format can update rows, so it can apply a stream of changes directly. The SQL statement MERGE matches incoming rows to existing rows by a key. It updates the rows that match and inserts the rows that do not. Doing both at once is called an upsert (update or insert). Teams often apply CDC changes to a lake table this way, every few minutes:

MERGE INTO ods_orders AS t
USING order_changes AS s            -- the newest change per order, from the binlog
ON t.order_id = s.order_id
WHEN MATCHED AND s.op = 'd' THEN DELETE
WHEN MATCHED THEN UPDATE SET status = s.status, updated_at = s.updated_at
WHEN NOT MATCHED AND s.op <> 'd' THEN
  INSERT (order_id, status, net_amount, updated_at)
  VALUES (s.order_id, s.status, s.net_amount, s.updated_at);

(The real table has more columns. This sketch shows the shape.)

How does a table format change one row, if files never change? There are two ways.

  • Copy-on-write. Rewrite each data file that holds a changed row. The new snapshot lists the new copy instead of the old file. Writing costs more. Reading stays fast, because readers see plain files. This is the same idea as Hive’s rewrite, but per file, not per folder, and with a commit and a snapshot.
  • Merge-on-read. Leave the old file alone. Write a small file that says which rows are deleted, plus a new file with the new versions. Readers merge them while they read. Writing is cheap. Reading costs more, until a compaction job merges the small files back into big ones.
Show the code 
bk.setup()
fig, ax = plt.subplots(figsize=(8, 5.0), layout="constrained")
ax.set_axis_off()
ax.set_xlim(-0.15, 9.85)
ax.set_ylim(-0.45, 8.75)


def snapshot(x, y, title, files):
    """A snapshot box listing its files; new files in tomato."""
    h = 0.62 * len(files) + 0.95
    ax.add_patch(FancyBboxPatch((x, y - h), 3.1, h, boxstyle="round,pad=0.02,rounding_size=0.12",
                                facecolor=bk.PAPER, edgecolor=bk.INK, linewidth=1.2))
    ax.text(x + 0.15, y - 0.38, title, fontsize=10.5, fontweight="semibold", color=bk.INK, va="center")
    for i, (name, new) in enumerate(files):
        fy = y - 0.95 - 0.62 * i
        ax.add_patch(FancyBboxPatch((x + 0.2, fy - 0.45), 2.7, 0.48, boxstyle="round,pad=0.01,rounding_size=0.06",
                                    facecolor=bk.TOMATO if new else "#ffffff", alpha=0.85 if new else 0.7,
                                    edgecolor=bk.INK, linewidth=0.8))
        ax.text(x + 0.35, fy - 0.21, name, fontsize=9, color=bk.INK, va="center",
                fontweight="semibold" if new else "normal")


def arrow(y):
    ax.add_patch(FancyArrowPatch((3.35, y), (6.35, y), arrowstyle="-|>", mutation_scale=16,
                                 color=bk.INK, linewidth=1.3))
    ax.text(4.85, y + 0.22, "refund one row", fontsize=9, color=bk.INK, ha="center")


base = [("f1.parquet", False), ("f2.parquet", False), ("f3.parquet", False)]
ax.text(0, 8.45, "Copy-on-write", fontsize=11.5, fontweight="semibold", color=bk.INK)
snapshot(0, 8.1, "Snapshot 41", base)
arrow(6.9)
snapshot(6.6, 8.1, "Snapshot 42", [("f1.parquet", False), ("f2-copy.parquet", True), ("f3.parquet", False)])
ax.text(0, 4.15, "Merge-on-read", fontsize=11.5, fontweight="semibold", color=bk.INK)
snapshot(0, 3.8, "Snapshot 41", base)
arrow(2.6)
snapshot(6.6, 3.8, "Snapshot 42", base + [("deletes: row 17 of f2", True), ("f4.parquet (new row)", True)])
plt.show()
Two rows of diagrams. In each row, a box labelled snapshot 41 lists three data files, an arrow labelled refund one row points to a box labelled snapshot 42. In the top row, copy-on-write, snapshot 42 lists the first and third files and a new tomato-coloured copy of the second file. In the bottom row, merge-on-read, snapshot 42 lists the three old files plus two small tomato files: a delete file and a file with the new row.
Figure 2: One refund in a table format. Each box is a snapshot: the list of files that make up the table. Old files stay on disk, so the old snapshot can still be read (time travel).

Time travel has a limit. Old snapshots keep old files on disk, and disk costs money. So teams delete old snapshots and their files on a schedule, and the time travel to those moments goes with them. (The box “Under the hood” gives the details for Iceberg and Delta Lake.)

Four table formats

A table format is this set of rules: how data files, metadata and snapshots are laid out and committed. It sits on top of a file format such as Parquet. Four open table formats are common today. All four give ACID commits, snapshots and time travel. They differ in where they put their effort.

Format Started at Best known for
Apache Iceberg Netflix Planning from metadata; safe schema changes
Delta Lake Databricks A simple log of numbered commits; close to Spark
Apache Hudi Uber Fast updates by record key; incremental queries
Apache Paimon Apache Flink (as Flink Table Store) Streaming updates from Flink

Iceberg keeps its metadata as a tree of files, so a query can find its data files without listing any folders. With hidden partitioning, the table works out partition values itself, such as the day of a timestamp.

Delta Lake keeps its history in a folder called _delta_log, with one numbered file per commit. It works most closely with Spark.

Hudi gives every record a record key and keeps an index to find it fast. It also answers incremental queries: “only the rows that changed since this moment.”

Paimon was built for streaming writes from Flink. It keeps tables with a key in an LSM tree (log-structured merge tree), which takes many small updates quickly and merges them later.

The lakehouse

Now look at the picture once more. The boathouse does not have its own water. It stands on posts in the lake. But inside, everything has a shelf and a place.

A lakehouse is that idea for data: lake storage, with warehouse behaviour on top. The files stay cheap and open, in object storage. A table format adds transactions, schemas, snapshots and time travel. A catalog adds names and permissions. Then the same copy of the data serves dashboards, SQL analysts, and machine learning, with no need to copy everything into a separate warehouse first.

The jars in the boathouse are still blank. Choosing what goes on each shelf, and how to label it, is the work of the next chapter.

Engines that read the lake

A table format is only a set of rules for files. To ask a question, you still need a query engine: a program that reads the files, runs the SQL, and returns the answer. Because the formats are open, one table can serve many engines. Spark (Chapter 10) and Flink (Chapter 14) read and write them. Three other names come up often.

Engine What it is Keeps its own data?
Trino A distributed SQL engine No: it reads data where it lives
ClickHouse A column-oriented database for fast analysis Yes, and it can read lake tables
Apache Doris, StarRocks MPP databases that speak MySQL’s protocol Yes, and they can read lake tables

Trino stores no data. Through connectors, one query can join a lake table with a table in a database such as MySQL.

ClickHouse keeps its own tables in its MergeTree storage engines. It can also read Iceberg and Delta Lake tables in place.

Apache Doris and StarRocks are MPP databases: massively parallel processing, where many machines each work on their share of one query. Both are popular for fast dashboards on fresh data. Both can read Hive tables and the main table formats.

A table as a list of files. Write \(F_v\) for the set of data files in snapshot \(v\), and \(D_v\) for the rows that its delete files remove. The table at snapshot \(v\) is every row in the files of \(F_v\), minus the rows in \(D_v\). A commit reads snapshot \(v\), writes new files, and proposes \(F_{v+1} = (F_v \setminus \text{removed}) \cup \text{added}\). It succeeds only if \(v\) is still the current snapshot. This is optimistic concurrency: assume no conflict, check at the end, and retry if someone else committed first.

Time travel, precisely. “As of time \(t\)” means the newest snapshot whose commit time is at or before \(t\). The commit time is when the change reached the table, not when the event happened in the world. Our replay of the binlog follows the same idea: it applies the changes in binlog order, which is commit order, and stops at a time. (Strictly, a binlog entry’s timestamp is when its statement started, a moment before the commit.) So “August as of 14 September” answers what did the table say then?, not what was true then? A chargeback on 25 September makes an August order refunded from 25 September on, in every “as of” view.

Iceberg’s metadata tree. Table metadata file (schema, partition spec, list of snapshots, current snapshot) → one manifest list per snapshot → manifest files (one entry per data file, with its partition values and column statistics) → data files. A commit writes a new metadata file and asks the catalog to swap its pointer from the old file to the new one. Format version 2 added delete files; version 3 added deletion vectors. Snapshots are expired by a maintenance procedure, which by default removes snapshots older than five days; it does not run by itself.

Delta Lake’s log. _delta_log/00000000000000000000.json, …01.json, and so on, zero-padded to 20 digits. Each file lists actions such as add file and remove file. Parquet checkpoint files summarise the log so far. VACUUM deletes data files that the current version no longer uses and that are older than the retention period (7 days by default), even if an older version still needs them; log entries are kept for 30 days by default (delta.logRetentionDuration). Time travel needs both the log and the data files.

Hudi’s two table types. Copy On Write: updates write new versions of the base files. Merge On Read: updates go to log files (often Avro, a row format) beside the base files; queries merge them, and compaction folds them into new base files.

Copy-on-write or merge-on-read, and cleaning up. Iceberg lets each table choose copy-on-write or merge-on-read for deletes, updates and merges; its default is copy-on-write. Delta Lake’s deletion vectors and Iceberg’s format version 3 mark deleted rows without rewriting the file, which is merge-on-read. Iceberg’s expiring snapshots removes old snapshots and the files that only they used; after that, time travel to them is gone.

Hive’s transactional tables. UPDATE and DELETE arrived in Hive 0.14 and MERGE in Hive 2.2. The table must be stored as ORC and marked transactional=true.

Who looks after the formats. Iceberg became a top-level Apache project in 2020, and Paimon in 2024; Delta Lake is hosted by the Linux Foundation. Hudi records every action on a table in its timeline. Paimon can also produce a complete changelog for streaming readers, if a table is set up for it.

The engines. Trino was called PrestoSQL until December 2020. ClickHouse reads lake tables with table functions such as iceberg() and deltaLake(). Doris began at Baidu; StarRocks grew from the Doris code and is a Linux Foundation project. Both reach lake tables through external catalogs: Doris for Hive, Iceberg, Hudi and Paimon (Delta Lake is experimental), StarRocks for all five.

Paimon’s LSM tree. Each bucket of a primary-key table keeps sorted runs of rows. New writes form small runs, and background compaction merges runs. When two rows have the same key, a merge engine decides the result: keep the latest (the default), fill in only the columns that changed (partial update), or add them up (aggregation).

Try it

The time machine. Move the slider to choose an “as of” day. The page shows August as the database showed it at the end of that day, and lists the August orders that changed after 14 September, up to that day.

Things to try:

  • Start at 25 September, the day of this chapter. Then move the slider back to 14 September: the change is zero, because that is the reference day.
  • Move to the right end. After 10 October, August stops changing.
  • Move to the left end, 31 August, the day August ended. Even then, refunds still had weeks to arrive.

The totals were replayed in advance from the full binlog, one row per day. The list of orders comes from the binlog itself, live in your browser. The two are separate calculations, and they agree.

Common traps

  • Comparing numbers with different “as of” times. August on 14 September and August today are two different numbers, and both are honest. Write the “as of” time next to every number you report.
  • Treating time travel as a backup. Snapshots live in the same storage as the table, and they are deleted on a schedule. A backup is a separate copy, kept somewhere else.
  • Reading “as of” as “what was true”. Time travel shows what the table said at a commit time. A late refund changes the past only from the day it arrives.
  • Committing too often. A stream that commits every few seconds creates many small files and many snapshots. Schedule compaction and snapshot expiry.
  • Two catalogs for one table. If two engines register the same files in two catalogs, each sees its own current snapshot. Choose one catalog as the source of truth.
TipAudit Instinct · As originally reported, and as restated

Accountants close a month. When a refund for an August sale arrives in September, it is normally booked in September. August’s ledger does not move. Only an error in August’s figures leads to a restatement: August is shown again, corrected, next to the figure as originally reported, with a note that explains the difference.

Steep’s orders table follows a different rule. It counts orders by the day they were placed, with their status today. So its August keeps moving for weeks. Neither rule is wrong, but a report must say which one it uses. Dana’s board will see August twice: in finance’s closed books, and in the warehouse. The two will differ, because one books each refund on the day it arrives and the other on the day of its order. With the binlog, Mia can explain the difference refund by refund.

Cut-off testing compares when a transaction really happened with the period in which it was recorded. Snapshots and binlog positions give the second date: they show what the table said, and when.

NoteInterview Corner

Q1. Why did teams move from plain Hive tables to Iceberg, Hudi or Delta Lake?

Plain Hive tables track only partitions. Changing a row means rewriting its partition, readers can see a rewrite half done, two writers can overwrite each other, and nothing records the old version. Planning a query means listing folders, which is slow on object storage. Table formats track every file in snapshot metadata. That gives ACID commits, row-level UPDATE, DELETE and MERGE (copy-on-write rewrites only the affected files; merge-on-read writes small delete or log files and merges them at read time), time travel and rollback, schema and partition evolution without rewriting data, file-level statistics for skipping, and built-in compaction. The formats are open, so many engines can share one table.

Q2. What is time travel useful for?

Reproducing a report exactly as it was first published, for an audit or a dispute. Debugging: compare today’s snapshot with yesterday’s to see what changed. Rolling back a bad write by making an older snapshot current again. Keeping a machine-learning training set reproducible. Its limits: it reaches back only as far as snapshots are kept, it follows commit time rather than business time, and it is not a backup.

Q3. Data lake, data warehouse, lakehouse: what is the difference?

A lake stores files of any kind in cheap storage, applies the schema when reading, and lets many engines share the data, but on its own it has no transactions and weak control over what the files mean. A warehouse stores managed tables, checks the schema on write, supports transactions and fast SQL, and governs access, but it costs more and keeps the data in its own format. A lakehouse puts a table format and a catalog on top of lake storage, so open files behave like warehouse tables: one copy of the data for dashboards and reports (business intelligence, or BI), SQL and machine learning.

That afternoon, Mia went back to her own numbers. If August had moved, the two weeks of the case had moved too. She replayed the binlog to the start of 14 September, the same moment as finance’s spreadsheet in Chapter 1, and counted the completed orders of those two weeks.

The week before: 45,627. The week of the drop: 43,494. A fall of 4.7%. Then she replayed to that afternoon: a fall of 4.9%. The week of the drop was newer, so fewer of its refunds had arrived yet. That is why the fall looked a little smaller at first.

“So the refunds moved your number,” said Theo.

“By a fraction of a point,” said Mia. “And they cannot touch Dana’s dashboard at all. It counts the app’s payment events. A refund does not delete an event.” She wrote the two numbers in her notebook, each with its date.

This book can look past that Friday. With every refund in, by 25 October, the end of the data, the fall is 5.0%: the final-status number this book has used since Chapter 1.

Reported change: −12.0% orders, the week of 7 September compared with the week before (CEO dashboard).

Explained so far: 0 of the 12 points. About 7 of them sit between the orders database and the dashboard (Clue 1).

Suspects: the app, and the service that writes to Kafka (Chapter 8). The iOS app, version 3.2.0, is the main suspect (Chapter 6). Not proved.

Ruled out: the matcha menu (Chapter 4), Kafka (Chapter 8), storage (Chapter 9), computing (Chapter 10), and now refunds and restatements. Late refunds take completed orders out of finance’s count, more of them from the newer week. So finance’s fall grows, from 4.7% at the start of 14 September to 4.9% that Friday afternoon, closer to the dashboard’s −12.0%. The gap between finance’s count and the dashboard shrinks: from 7.3 to 7.1 points. The dashboard counts payment events, which refunds never remove. (By 25 October, the end of the data, that gap is 7.0 points: refunds moved it by less than half a point. They cannot explain 7.)

Open questions: Which orders have no order_completed event, and why? Did iOS 3.2.1, released on 24 September, fix it? (Chapter 15.)

New evidence: 192 August orders were refunded between 14 September and that Friday afternoon. A table that remembers can answer “as of when?” for any number. New rule: every number gets an “as of”.

Recap

  • A data lake keeps cheap, open files of any kind; a data warehouse keeps managed tables with transactions. Plain Hive tables on a lake cannot update a row, cannot commit safely, and forget every old version.
  • Table formats (Iceberg, Delta Lake, Hudi, Paimon) keep data files unchanged and add a log of snapshots. That gives ACID commits, MERGE, schema evolution and time travel. Lake storage with this warehouse behaviour is a lakehouse.
  • Data changes after the fact: by 25 October, the end of the data, 300 August orders had been refunded after 14 September. Every number needs an “as of” time, and a table that remembers can reproduce it.
English 中文
data lake 数据湖
data warehouse 数据仓库
lakehouse 湖仓一体
data swamp 数据沼泽
chargeback 拒付 / 退单
table format 表格式
metadata 元数据
snapshot 快照
commit 提交
catalog 目录
ACID 事务特性（原子性、一致性、隔离性、持久性）
time travel 时间旅行
schema evolution 模式演进
upsert / merge 更新插入 / 合并
copy-on-write 写时复制
merge-on-read 读时合并
compaction 合并 / 压实
deletion vector 删除向量
query engine 查询引擎
MPP (massively parallel processing) 大规模并行处理
restatement 重述
as originally reported 原报告数
cut-off testing 截止性测试

Further reading