The problem: plain files have no ACID

A data lake built on raw Parquet or ORC files has no built-in concept of a transaction — two concurrent writers can corrupt a dataset, and a reader can see a half-written, inconsistent state mid-write. Open table formats sit on top of those same files and add a metadata layer that tracks which files make up a consistent snapshot of the table at any given time, effectively giving a folder of files the reliability guarantees of a database table.

Advertisement

Time travel across the three formats

time-travel-syntax.sql
-- Delta Lake: query a table as of a previous version number
SELECT * FROM orders VERSION AS OF 12;

-- Delta Lake: query a table as of a previous timestamp
SELECT * FROM orders TIMESTAMP AS OF '2026-08-01';

-- Iceberg: query a table's snapshot history
SELECT * FROM orders.snapshots;

-- Iceberg: query as of a specific snapshot ID
SELECT * FROM orders FOR SYSTEM_VERSION AS OF 4834203495834;

Each format retains references to previous snapshots in its metadata rather than only the current state, which is what makes both auditing past changes and recovering from an accidental bad write possible without a separate backup system.

Advertisement

Key differences

  • Delta Lake originated with Databricks and has the deepest, most mature integration with Spark and Databricks-native tooling.
  • Apache Iceberg was designed engine-agnostic from the outset, with broad support across many different query engines beyond any single vendor's ecosystem.
  • Apache Hudi emphasizes efficient upsert operations and incremental/streaming ingestion patterns as a core design focus.

Side-by-side comparison

AspectDelta LakeApache IcebergApache Hudi
OriginDatabricksNetflix (now Apache)Uber (now Apache)
Ecosystem fitDeepest with Spark/DatabricksBroad, engine-agnosticStrong for streaming/upserts
Core strengthMature tooling, wide adoptionMulti-engine flexibilityIncremental ingestion efficiency

Adoption guidance

The right format usually follows from the query engine and workload pattern already in place rather than an abstract feature comparison: teams standardized on Databricks and Spark often default to Delta Lake for its native integration, teams wanting flexibility across multiple query engines often favor Iceberg, and teams with heavy upsert or streaming-ingestion requirements often lean toward Hudi.

Key takeaways

  • Open table formats add ACID transactions and time travel to plain data lake files via a metadata layer.
  • Delta Lake, Iceberg, and Hudi all support standard SQL access, including MERGE-based upserts and format-specific time-travel syntax.
  • Delta Lake integrates deepest with Spark/Databricks; Iceberg is engine-agnostic; Hudi focuses on efficient upserts and streaming ingestion.
  • Choice usually follows from the existing query engine and workload pattern, not an isolated feature comparison.