| ID | product.processing.batch_processor.spark_runner |
|---|---|
| Description | Submits and supervises Spark jobs. |
| Key | spark_runner |
| Type | component |
| Team | Stream Processing |
| Owner | stream-processing@iotgateway.example |
| Status | active |
| Technologies | Scala, Apache Spark |
| Attributes | executor_memory: 8g, parallelism: 200 |
| Tags |
Table of contents
Guide
Spark Runner
The Spark Runner is the execution engine of the batch side. It takes a job dispatched by the Job Scheduler, submits a Spark application, and supervises it until the curated output is handed to the Export Writer.
Elastic by design
Executors scale with the size of the day being processed. A normal day uses a modest pool; a backfill of weeks spins up far more parallelism and releases it when the job completes.