Daily batch refresh
| ID | pipelines.daily-batch |
|---|---|
| Description | Orchestrator runs the daily batch pipelines that refresh the warehouse. |
| Key | daily-batch |
| Tags | processing storage |
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
100%
Steps
Workflow steps
| Number | Source | Destination | Description |
|---|---|---|---|
| 1 | Orchestrator | DAG scheduler | Trigger daily DAG |
| 2 | DAG scheduler | Batch pipeline | Run tasks in order |
| 3 | Batch pipeline | dbt engine | Transform raw to curated |
| 4 | dbt engine | Data validator | Run tests |
| - if Tests pass | |||
| 5 | Data validator | Warehouse loader | Materialize tables |
| 6 | Warehouse loader | Data warehouse | Load gold layer |
| else Tests fail | |||
| 7 | Data validator | Orchestrator | Block downstream |
| end | |||