Stream Pipeline

ID	product.processing.stream_pipeline
Description	Stateful Flink / Spark Streaming jobs.
Key	stream_pipeline
Type	microservice:service
Status	active
Team	Data Engineering
Owner	eng@dataplatform.example
Technologies	Spark, dbt, Airflow
Tags

Table of contents

Links and Communications

Requirements

Attributes

engine	Flink
mode	streaming
state	keyed

Guide

Stream Pipeline

Stateful Flink / Spark Streaming jobs.

Container

Part of the Processing system. Components inside this container link only to siblings — cross-container integration happens at the service boundary.

Components

Could not build diagram for model "example-data-processing".

Flink jobs

Links and Communications

2 incoming links · 1 workflow · 1 async channel

Incoming Links(2)

Source object	Label	Link type	Relation	Raw id	Required	Description
Stream Router example-data-processing.product.ingestion.stream_router microservice:service	feeds	—	links	product.processing.stream_pipeline	—	—
Orchestrator example-data-processing.product.processing.orchestrator microservice:service	—	—	links	product.processing.stream_pipeline	—	—

Async links(1)

Produces(1)

Channel	Broker	Capture	Consumer
Domain events domain.events · topic	Event Bus external_systems.event_bus	—	Kafka Consumer

Channel

Broker

Capture

Consumer

Domain events

domain.events · topic

Event Bus external_systems.event_bus

—

Kafka Consumer

Related Workflows(1)

Workflow	Model	Participant id	Participant label
Real-time event ingestion streaming.realtime-event	example-data-processing	product.processing.stream_pipeline	Stream pipeline