Storage is where data lives between ingress and consumption. It separates raw history (the bronze lake on object storage) from curated analytics tables (the gold warehouse) and holds the metadata — schemas, table catalog entries and snapshot lineage — that every other system references.
Responsibility in one sentence
Persist raw and curated datasets durably, manage partitions and snapshots, and expose authoritative schema and catalog metadata to pipelines and query engines.
Could not build diagram for model "example-data-processing".
Bronze/silver Iceberg tables on object storage — partitioned Parquet with compaction.
Columnar OLAP gold layer — curated tables with time-travel snapshots.
Schema registry, table catalog and version compatibility rules.
The Data Lake accepts append-only writes from Ingestion. The Lake Writer partitions by time/key; the Compaction Job keeps query performance healthy.
CDC events land in the bronze layer as part of the sync flow:
Curated gold tables are materialized after the nightly batch completes: