Daily batch refresh
| ID | pipelines.daily-batch |
|---|---|
| Description | Orchestrator runs the daily batch pipelines that refresh the warehouse. |
| Key | daily-batch |
| Tags | processing storage |
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
100%
Specification
Daily batch refresh
Orchestrator runs the daily batch pipelines that refresh the warehouse.
End-to-end flow
This workflow is documented in specs under Workflows and embedded here as the canonical sequence diagram for the model.
Steps
Workflow steps
| Number | Source | Destination | Description |
|---|---|---|---|
| 1 | Orchestrator | DAG scheduler | Trigger daily DAG |
| 2 | DAG scheduler | Batch pipeline | Run tasks in order |
| 3 | Batch pipeline | dbt engine | Transform raw to curated |
| 4 | dbt engine | Data validator | Run tests |
| - if Tests pass | |||
| 5 | Data validator | Warehouse loader | Materialize tables |
| 6 | Warehouse loader | Data warehouse | Load gold layer |
| else Tests fail | |||
| 7 | Data validator | Orchestrator | Block downstream |
| end | |||