The Kafka Consumer is the entry point of the streaming pipeline. It reads the validated telemetry topics published by ingestion and hands records to the Window Aggregator with exactly-once semantics.
Offsets and checkpoints
Consumer offsets are committed as part of Flink checkpoints, not independently. On recovery the pipeline rewinds to the last checkpoint so no window is double-counted and none is lost.
The consumer subscribes to the telemetry topics, spread across partitions keyed by device id so each device’s events stay ordered.
Records are decoded and lightly validated; malformed payloads are routed to a dead-letter topic instead of stalling the pipeline.
Valid events are forwarded downstream, preserving event-time timestamps for correct windowing.