| ID | product.processing.batch_pipeline.spark_runner |
|---|---|
| Description | Submits and supervises Spark jobs on the cluster. |
| Key | spark_runner |
| Type | component |
| Status | active |
| Team | Data Engineering |
| Owner | eng@dataplatform.example |
| Tags |
Table of contents
Attributes
| cluster | YARN/K8s |
|---|---|
| mode | batch |
| shuffle | optimized |
Guide
Spark Runner
Submits and supervises Spark jobs on the cluster.
Component
Internal module inside Batch Pipeline. See the parent container guide for the end-to-end flow this component participates in.