| ID | product.processing |
|---|---|
| Description | Batch and streaming transformations orchestrated end-to-end. |
| Key | processing |
| Type | system |
| Team | Data Engineering |
| Owner | eng@dataplatform.example |
| Status | active |
| Technologies | Spark, dbt, Airflow |
| Attributes | modes: batch,stream, orchestration: dag |
| Tags |
Table of contents
Guide
Processing
Processing is where raw data becomes trustworthy. It runs the orchestrated batch DAGs that refresh gold warehouse tables every night, and the stateful streaming jobs that power live dashboards — always with validation gates that block corrupt data from propagating downstream.
Responsibility in one sentence
Schedule and execute batch and streaming transformations, validate outputs, emit lineage, and halt dependent work when quality rules fail.