SpecsModelExploreRequirements
SpecsModelExploreRequirements
Processing
Overview
Guide
Twin streams
Links and Communications
IDproduct.processing
DescriptionBatch and streaming transformations orchestrated end-to-end.
Keyprocessing
Typesystem
Statusactive
TeamData Engineering
Owner
eng@dataplatform.example
TechnologiesSpark, dbt, Airflow
Tags
Table of contents
Guide
Processing
System at a glance
Containers
Two paths for one platform
Responsibilities
Related workflows
Links and Communications
Requirements

Attributes

modesbatch,stream
orchestrationdag

Guide

Processing

Processing is where raw data becomes trustworthy. It runs the orchestrated batch DAGs that refresh gold warehouse tables every night, and the stateful streaming jobs that power live dashboards — always with validation gates that block corrupt data from propagating downstream.

Responsibility in one sentence

Schedule and execute batch and streaming transformations, validate outputs, emit lineage, and halt dependent work when quality rules fail.

System at a glance

Could not build diagram for model "example-data-processing".

Containers

Orchestrator

DAG scheduler — triggers, retries, SLA monitoring and task coordination.

Batch Pipeline

Spark/dbt jobs that transform bronze lake tables into gold warehouse tables.

Stream Pipeline

Stateful Flink jobs — windowed aggregates, enrichment and late-data handling.

Two paths for one platform

Responsibilities

  • DAG scheduling, task execution, retry and SLA breach alerting.
  • dbt/SQL transforms, Spark jobs and output validation (Great Expectations).
  • Stateful stream processing with watermarks and compensating late-data updates.
  • Dataset-level lineage emission on every successful run.

Related workflows

The nightly refresh is owned end-to-end by batch processing:

Real-time aggregates for dashboards flow through the stream pipeline:

Quality gates

When the Data Validator fails a test, the orchestrator blocks downstream tasks — the batch path stops before gold tables are corrupted.

Links and Communications

2 incoming links · 2 outgoing links

Incoming Links(2)

Source objectLabelLink typeRelationRaw idRequiredDescription
Data Engineer

example-data-processing.actors.data_engineer

actor
builds pipelines in—linksproduct.processing——
Storage

example-data-processing.product.storage

system
feeds—linksproduct.processing——

Outgoing Links(2)

Target objectLabelLink typeRelationRaw idRequiredDescription
Serving

example-data-processing.product.serving

system
publishes results to—linksproduct.serving——
Governance

example-data-processing.product.governance

system
reports lineage to—linksproduct.governance——

Requirements