CDC sync from production
| ID | ingestion.cdc-sync |
|---|---|
| Description | Replicate operational DB changes to the lake in near real time. |
| Key | cdc-sync |
| Tags | ingestion storage |
100%
Specification
CDC sync from production
Replicate operational DB changes to the lake in near real time.
End-to-end flow
This workflow is documented in specs under Workflows and embedded here as the canonical sequence diagram for the model.
Steps
Workflow steps
| Number | Source | Destination | Description |
|---|---|---|---|
| 1 | Production DB | CDC reader | WAL/binlog change |
| 2 | CDC reader | Position tracker | Persist offset |
| 3 | CDC reader | Kafka publisher | Emit event |
| 4 | Kafka publisher | Event bus | Publish |
| 5 | Event bus | Stream router | Consume |
| 6 | Stream router | Data lake | Write bronze layer |