Processing is where raw data becomes trustworthy. It runs the orchestrated batch DAGs that refresh gold warehouse tables every night, and the stateful streaming jobs that power live dashboards — always with validation gates that block corrupt data from propagating downstream.
Responsibility in one sentence
Schedule and execute batch and streaming transformations, validate outputs, emit lineage, and halt dependent work when quality rules fail.
Could not build diagram for model "example-data-processing".
DAG scheduler — triggers, retries, SLA monitoring and task coordination.
Spark/dbt jobs that transform bronze lake tables into gold warehouse tables.
Stateful Flink jobs — windowed aggregates, enrichment and late-data handling.
The nightly refresh is owned end-to-end by batch processing:
Real-time aggregates for dashboards flow through the stream pipeline:
Quality gates
When the Data Validator fails a test, the orchestrator blocks downstream tasks — the batch path stops before gold tables are corrupted.