.svg)
Data teams running Apache Spark on Kubernetes often miss the runtime signals that degrade job performance, delay pipelines, and inflate cloud costs. This guide breaks down five commonly overlooked Spark-on-Kubernetes observability blind spots, along with actionable fixes and a quick-wins checklist to keep production workloads stable.
A Spark job stalls sometime around Saturday night. By Monday morning, the analytics team is asking why dashboards stopped updating, even though the job still shows as “running” in the Spark UI. That is what makes Spark on Kubernetes failures so frustrating.
Spark sees application state, Kubernetes sees container state, and neither tells the full story. Executor restarts, Pending backlogs, heartbeat lag, shuffle bottlenecks, and memory overhead issues build for hours while the cluster appears healthy from the application layer, exposing gaps in traditional end-to-end data quality monitoring.
The five signals discussed in this blog are the operational blind spots most Spark-on-Kubernetes teams miss. We've also included a quick-wins checklist to diagnose and fix Spark issues faster.
Why Teams Miss Critical Signals
The Spark UI isn’t broken; it is scoped to the application layer. It tracks stages, tasks, shuffle partitions, and executor behavior. What it doesn’t tell you is that an executor pod restarted three times mid-stage, or that half your executors sat in Pending for ten minutes before the scheduler placed them.
Kubernetes fills part of that gap with pod phases, restart counters, and scheduler events, but it has no visibility into heartbeat intervals, shuffle pressure, or driver-side task metrics. kube-state-metrics helps expose Kubernetes object state, yet the signals remain fragmented across Spark and Kubernetes.
The signals already exist across both systems. The problem is that most Spark cluster setups lack the correlation layer between Spark and Kubernetes. That disconnect is where Spark observability breaks down.
The following five signals expose the operational impact of that fragmentation.
.
Signal #1: Silent Executor Restarts
Executor pods restart without a trace. Kubernetes brings them back, Spark re-registers them, and the Spark UI shows no obvious error. What you lose is time, task retries, and wall-clock job duration.
Diagnosing this issue requires visibility into both Spark and Kubernetes events:
1. Start with restart counts
Monitor kube_pod_container_status_restarts_total from kube-state-metrics. A stable restart counter is normal, while rising restart counts across the same executor pods usually signal container instability.
Pair this with kubectl get pods -n <ns> -l spark-role=executor to inspect the RESTARTS column directly.
2. Cross-reference with Spark activity
Use the Spark History Server to compare restart spikes with executor add/remove patterns. This helps separate normal autoscaling from infrastructure-level container instability.
3. Alert on sustained restart patterns
For alerting, apply a Prometheus rate() against the restart counter with a sustained for clause to avoid false pages from transient node events.
The table below shows how these patterns map to investigation steps:
Signal #2: Lagging Driver Heartbeats
Executors continuously send heartbeat signals to the driver to confirm they are alive and to share in-progress task metrics.
The key configuration pair here is spark.executor.heartbeatInterval and spark.network.timeout. When heartbeat signals start lagging, the driver marks executors as lost even before any container actually restarts.
The tricky part here is that heartbeat degradation does not always start at the executor layer. In many cases, a CPU-throttled driver pod struggles to process incoming heartbeats, creating symptoms that look like executor instability. To surface the root cause:
- Scrape Spark’s REST and metrics endpoints in Prometheus format
- Watch executor expiry patterns alongside driver CPU pressure in your Kubernetes cluster dashboard
- Alert on sustained heartbeat degradation, not isolated events
- Keep spark.executor.heartbeatInterval well below spark.network.timeout
Pro tip: Before tuning executors in Spark Kubernetes deployments, check whether the driver pod has enough CPU and memory to process heartbeats reliably.
Signal #3: Pod Pending Spikes
Brief Pending time at job startup is expected. A growing backlog of executor pods stuck in Pending is not. It means your jobs are burning wall-clock time before a single task even runs.
Pending states almost always point to a scheduling constraint. Since Kubernetes schedules on resource requests, not actual utilization, a node can appear half-idle and still reject a pod if the declared requests exceed allocatable capacity.
Common causes include:
- Namespace-level ResourceQuota limits blocking new pods
- Taint and toleration mismatches
- Node affinity rules that no longer match after node pool rotation
To diagnose and monitor the issue:
- Run kubectl describe pod <pod> -n <ns> and inspect the Events section to identify the scheduling constraint.
- Track kube_pod_status_phase{phase="Pending"} and establish a baseline for acceptable startup delay.
- Alert when Pending duration consistently exceeds 2–3x your normal startup window.
- Validate ResourceQuota availability and node capacity regularly to avoid hidden scheduling bottlenecks.
Signal #4: Choked Shuffle I/O
Shuffle slowdowns often originate below the application layer itself. In Spark-on-Kubernetes deployments, two infrastructure-level factors directly affect shuffle performance: the storage backing spark.local.dir and the network policies governing executor-to-executor traffic.
Track shuffle read/write metrics via Spark’s Prometheus-formatted endpoints at runtime. A sustained increase relative to your workload baseline, not a one-off spike, is the signal worth investigating.
Two Kubernetes-specific factors matter here:
- spark.local.dir defaults to /tmp. If that path maps to a network-backed volume, shuffle spill becomes a bottleneck almost immediately.
- Kubernetes NetworkPolicies require explicit ingress and egress rules. Restrictive namespace policies that block executor pod-to-pod TCP traffic can quietly throttle shuffle fetches without producing a clear error.
Note: Spark’s KubernetesLocalDiskShuffleDataIO plugin is a documented option for Kubernetes-aware shuffle I/O behavior. Benchmark it carefully against your storage class and CNI before enabling it in production.
Signal #5: MemoryOverhead Creep
Apache Spark observability on Kubernetes introduces a memory accounting problem that does not exist in the same form on other schedulers.
spark.executor.memoryOverhead and spark.executor.memoryOverheadFactor define non-heap allocation for native overheads, while spark.kubernetes.memoryOverheadFactor extends that further for tmpfs-backed local directories.
The symptom is gradual. Memory usage steadily climbs toward the container limit without a sharp spike. Common contributors include:
- Native library allocations and off-heap buffer growth
- PySpark subprocesses. Spark explicitly documents that unconfigured Python memory is unbounded and competes with everything else in the container
Monitor both container memory and Spark executor metrics carefully, because confusing JVM heap growth with non-heap overhead often leads to the wrong fix.
If the trend is sustained, investigate in this order:
- Increase the overhead factor where non-heap growth is confirmed
- Bound PySpark memory explicitly if Python workloads are involved
- Reconsider tmpfs-backed spark.local.dir given its pod-level memory accounting implications
Quick Wins Checklist
Healthy ranges and alert thresholds are intentionally baseline-relative because Kubernetes and Prometheus practices vary significantly across Spark-on-Kubernetes environments.
The table below maps each signal to its key metric, healthy pattern, alert trigger, diagnostic method, and first remediation step.
Pull this into your next sprint stand-up or incident retrospective. Alertmanager's grouping, inhibition, and silences are what keep these signals actionable rather than noisy.
Building a Proactive Observability Culture
A checklist only works when it becomes part of day-to-day operations. Start with a few practical changes:
- Configure sustained-condition alert rules using Prometheus for clauses
- Build runbooks that map each signal to a clear diagnostic and escalation path
- Run periodic health reviews against historical baselines with automated anomaly detection to catch drift early
- Finally, route alerts through Alertmanager with grouping and inhibition to prevent restart storms from generating dozens of pages during a single cluster event
Achieving reliable Spark cluster observability requires correlating driver and executor pod events, restart counters, scheduling failures, and Spark application metrics in one place. That is exactly what Acceldata xLake is built for.
Instead of context-switching across multiple tools, teams get driver and executor logs, OOMKill signals, spot eviction reasons, and scheduling failures in a single control plane for Spark on Kubernetes and EKS observability.
If you're still reacting to Spark failures, book a personalized demo today and see how Acceldata xLake unifies your Spark and Kubernetes data observability in one control plane.
Spark-on-Kubernetes Monitoring: Frequently Asked Questions
How can I detect executor out-of-memory failures before jobs crash?
Track non-heap pod memory growth relative to your configured spark.executor.memoryOverhead. Account for PySpark processes and tmpfs-backed scratch directories, which share the same non-JVM overhead budget and can silently consume it before a crash occurs.
What metrics stack is recommended for comprehensive Spark on Kubernetes observability?
Combine Spark's monitoring surfaces (UI, event logs, REST/metrics) with kube-state-metrics for Kubernetes object state. Operationalize alerting using Prometheus and Alertmanager for routing, grouping, and silencing.
How do missed monitoring signals impact cloud costs specifically?
Pending pods extend wall-clock runtime even when node utilization appears low because Kubernetes schedules on requests, not real usage. Missed incidents mean delayed remediation and compute spend continuing on stalled or degraded jobs.
What distinguishes pod restarts from driver failures in terms of monitoring approach?
Pod restarts are Kubernetes container lifecycle events observable via restart counters. Driver failures manifest through heartbeat and network timeout behavior per Spark's configuration model. Accurate classification requires signals from both layers simultaneously.
How can I test alert rules for these signals in a staging environment?
Use Prometheus for durations to distinguish transient from sustained conditions by inducing short-lived versus persistent faults. Alertmanager silences let you mute pages during controlled tests while validating that grouping and inhibition work as designed before pushing to production.
.webp)
.webp)
