| ID | pipelines.daily-batch |
|---|---|
| Description | Orchestrator runs the daily batch pipelines that refresh the warehouse. |
| Key | daily-batch |
| Type | Sequence Workflow Diagram |
| Tags | processing storage |
Orchestrator runs the daily batch pipelines that refresh the warehouse.
End-to-end flow
This workflow is documented in specs under Workflows and embedded here as the canonical sequence diagram for the model.
| Number | Source | Destination | Description |
|---|---|---|---|
| 1 | Orchestrator | DAG scheduler | Trigger paths./dags/{dagId}/runs.post on dest |
| 2 | DAG scheduler | Batch pipeline | Run tasks paths./runs/{runId}/tasks.get on dest |
| 3 | Batch pipeline | dbt engine | Transform raw to curated on dest |
| 4 | dbt engine | Data validator | Run tests on dest from source |
| - if Tests pass | |||
| 5 | Data validator | Warehouse loader | Materialize tables via dest |
| 6 | Warehouse loader | Data warehouse | Load gold tables.gold_orders |
| else Tests fail | |||
| 7 | Data validator | Orchestrator | Block downstream dest |
| end | |||