Data warehouse: schema-on-write

A schema is defined before any data lands, and every row is validated against it on the way in. This makes queries fast and predictable, at the cost of flexibility — a new, unexpected field from a source system has nowhere to go until the schema is explicitly changed:

warehouse-schema-on-write.sql
CREATE TABLE orders (
  order_id    INT PRIMARY KEY,
  customer_id INT,
  order_date  DATE,
  amount      DECIMAL(10,2)
);
-- A row missing 'amount' or with a non-numeric amount is rejected at load time,
-- not discovered later during a query
Advertisement

Data lake: schema-on-read

Files land in object storage (S3, ADLS, GCS) in whatever format they arrive — JSON, CSV, Parquet, images, logs — with no schema enforced at write time. Schema is applied only when a query engine reads the data:

lake-schema-on-read.sql
-- Spark SQL reading raw JSON files directly from lake storage,
-- inferring structure at query time rather than at write time
CREATE TEMP VIEW raw_orders USING json
OPTIONS (path 's3://data-lake/raw/orders/*.json');

SELECT * FROM raw_orders LIMIT 10;

This is flexible — any file, any structure, no upfront modeling — but that same flexibility is exactly what turns into a "data swamp" without deliberate organization: undocumented files nobody can confidently query or trust.

Lakehouse: open table formats bridge the gap

lakehouse-delta-example.sql
-- Databricks SQL / Delta Lake: ACID MERGE directly on lake-stored files
MERGE INTO lakehouse.orders AS target
USING staged_updates AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET target.amount = source.amount
WHEN NOT MATCHED THEN INSERT (order_id, customer_id, order_date, amount)
  VALUES (source.order_id, source.customer_id, source.order_date, source.amount);

-- Time travel: query the table exactly as it existed yesterday
SELECT * FROM lakehouse.orders VERSION AS OF 42;

Open table formats (Delta Lake, Apache Iceberg, Apache Hudi) add a metadata layer over plain files in lake storage, tracking schema, transactions, and file versions — enabling ACID guarantees, schema enforcement, and time travel that raw files alone never had, all while the underlying data still lives in cheap, open object storage rather than a proprietary warehouse format.

Advertisement

Side-by-side comparison

Data WarehouseData LakeLakehouse
SchemaOn writeOn readEnforced via table format
Data typesStructured onlyAny formatAny format, structured access
ACID transactionsYes, nativeNoYes, via open table format
Storage costHigherLow (object storage)Low (object storage)
Best forBI, structured reportingRaw retention, ML, explorationBoth, on one platform
ExamplesSnowflake, Redshift, BigQueryS3/ADLS + Spark/HiveDatabricks, Iceberg on any engine

Which one for which workload

  • Structured BI reporting with predictable, fast queries → data warehouse, or a warehouse-shaped Gold layer inside a lakehouse.
  • Large volumes of raw, varied-format data, machine learning training sets, exploratory analysis → data lake or lakehouse.
  • One platform needs to serve both BI and ML from the same underlying data → lakehouse, which is exactly the gap it was designed to close.

Common mistakes

  • Dumping files into lake storage with no naming convention, metadata, or catalog, turning a data lake into an unsearchable data swamp.
  • Assuming a lakehouse automatically eliminates the need for any warehouse-style layer — most real lakehouses still build Gold-layer tables that behave like warehouse tables.
  • Choosing a platform because of industry hype rather than the actual shape of the workload — structured BI vs. varied-format ML data genuinely need different things.
  • Treating "lakehouse" as a single product rather than a pattern — it's implemented differently across Databricks, Snowflake's Iceberg support, and open-source engines on raw Iceberg/Delta tables.

Key takeaways

  • Data warehouse: schema-on-write, structured, fast for BI, higher storage cost.
  • Data lake: schema-on-read, any file format, cheap, needs deliberate organization to avoid becoming a swamp.
  • Lakehouse: adds ACID transactions and schema enforcement on lake storage via an open table format.
  • Most real platforms combine elements of all three rather than picking exactly one.