SpecsModelExploreRequirements
SpecsModelExploreRequirements
Spark Runner
Overview
Guide
Twin streams
Links and Communications
IDproduct.processing.batch_processor.spark_runner
DescriptionSubmits and supervises Spark jobs.
Keyspark_runner
Typecomponent
Statusactive
TeamStream Processing
Owner
stream-processing@iotgateway.example
TechnologiesScala, Apache Spark
Tags
Table of contents
Guide
Spark Runner
Execution
Notes
Links and Communications
Requirements

Attributes

executor_memory8g
parallelism200

Guide

Spark Runner

The Spark Runner is the execution engine of the batch side. It takes a job dispatched by the Job Scheduler, submits a Spark application, and supervises it until the curated output is handed to the Export Writer.

Elastic by design

Executors scale with the size of the day being processed. A normal day uses a modest pool; a backfill of weeks spins up far more parallelism and releases it when the job completes.

Execution

The runner builds the application config — input partitions, executor memory, parallelism — and submits it to the cluster.

Notes

  • Reads raw telemetry for the target date and produces curated, deduplicated records with late data folded in.
  • Resource limits are bounded so a large backfill cannot starve scheduled daily runs.

Links and Communications

1 incoming link · 1 outgoing link

Incoming Links(1)

Source objectLabelLink typeRelationRaw idRequiredDescription
Export Writer

example-iot.product.processing.batch_processor.export_writer

component
writes datasets from—linksproduct.processing.batch_processor.spark_runner——

Outgoing Links(1)

Target objectLabelLink typeRelationRaw idRequiredDescription
Job Scheduler

example-iot.product.processing.batch_processor.job_scheduler

component
submits jobs to—linksproduct.processing.batch_processor.job_scheduler——

Requirements