SpecsModelExploreRequirements
SpecsModelExploreRequirements
Daily batch refresh
Overview
Specification
Steps
Workflow steps
NumberSourceDestinationDescription
1OrchestratorDAG schedulerTrigger paths./dags/{dagId}/runs.post on dest
2DAG schedulerBatch pipelineRun tasks paths./runs/{runId}/tasks.get on dest
3Batch pipelinedbt engineTransform raw to curated on dest
4dbt engineData validatorRun tests on dest from source
- if Tests pass
5Data validatorWarehouse loaderMaterialize tables via dest
6Warehouse loaderData warehouseLoad gold tables.gold_orders
else Tests fail
7Data validatorOrchestratorBlock downstream dest
end