| ID | product.processing.batch_processor |
|---|---|
| Description | Daily Spark jobs producing curated datasets. |
| Key | batch_processor |
| Type | microservice:service |
| Team | Stream Processing |
| Owner | stream-processing@iotgateway.example |
| Status | active |
| Technologies | Scala, Apache Spark, S3 |
| Attributes | cadence: daily, parallelism: 200 |
| Tags |
Table of contents
Guide
Batch Processor
The Batch Processor complements the real-time pipeline. Where the stream side optimises for freshness, the batch side optimises for completeness: it reprocesses full days of telemetry into curated, columnar datasets that power analytics and long-term archival.
Why batch as well as stream
Streaming answers “what is happening now?”; batch answers “what really happened?” — with late data folded in, corrections applied, and results written in an analytics-friendly format.