| ID | product.processing.batch_processor.export_writer |
|---|---|
| Description | Writes curated datasets to object storage. |
| Key | export_writer |
| Type | component |
| Team | Stream Processing |
| Owner | stream-processing@iotgateway.example |
| Status | active |
| Technologies | Scala, S3 |
| Attributes | compression: snappy, format: Parquet |
| Tags |
Table of contents
Guide
Export Writer
The Export Writer is the final stage of a batch job. It takes the curated records produced by the Spark Runner and writes them as partitioned, columnar files into object storage for analytics and long-term archival.
Write format
Output is Parquet with snappy compression, partitioned by date. The columnar layout keeps downstream analytical scans cheap and the partitioning lets consumers prune to the days they need.