Daily batch refresh
Workflow steps
| Number | Source | Destination | Description |
|---|---|---|---|
| 1 | Orchestrator | DAG scheduler | Trigger daily DAG |
| 2 | DAG scheduler | Batch pipeline | Run tasks in order |
| 3 | Batch pipeline | dbt engine | Transform raw to curated |
| 4 | dbt engine | Data validator | Run tests |
| - if Tests pass | |||
| 5 | Data validator | Warehouse loader | Materialize tables |
| 6 | Warehouse loader | Data warehouse | Load gold layer |
| else Tests fail | |||
| 7 | Data validator | Orchestrator | Block downstream |
| end | |||