| ID | product.processing.batch_pipeline.spark_runner |
|---|---|
| Description | Submits and supervises Spark jobs on the cluster. |
| Key | spark_runner |
| Type | component |
| Team | Data Engineering |
| Owner | eng@dataplatform.example |
| Status | active |
| Attributes | cluster: YARN/K8s, mode: batch, shuffle: optimized |
| Tags |
Table of contents
Guide
Spark Runner
Submits and supervises Spark jobs on the cluster.
Component
Internal module inside Batch Pipeline. See the parent container guide for the end-to-end flow this component participates in.