Spark : Cancelling potential speculative or zombie tasks for this job

 

>PROBLEM

Running a long iteration, Spark was returning messages like these:

[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - ResultStage 433 (save at PostgresRepository.java:64) finished in 328 ms

[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - Job 433 is finished. Cancelling potential speculative or zombie tasks for this job

[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Canceling stage 433

[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Killing all running tasks in stage 433: Stage finished

>SOLUTION

It was due to the SparkSession’s memory leak.

A zombie remembers C/C++ programming when the reference is lost, or Java programming when the resource is not closed.

The SparkSession must be treated like a Java resource.

Rule of thumb: 

Make sure that you are not returning any Spark’s object after closing a SparkSession.

TIP:

If it is necessary to return data among different sessions, convert to a POJO.
For instance, Dataset<>  to   MyPojo.

>ENV

Spark 2.x

Java 17

eclipse 2025-03: JBoss Tools 4.29.1 issue: An internal error occurred during: "Importing Maven projects". class org.eclipse.jdt.internal.core.JavaNature cannot be cast to class org.eclipse.jdt.core.IJavaProject (org.eclipse.jdt.internal.core.JavaNature and org.eclipse.jdt.core.IJavaProject are in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @21fd7397)

>CONTEXT

Debian 11, Eclipse 2025-03/Java21 and project using OpenJDK 17.

>PROBLEM

After the installation of JBoss Tools 4.29.1, as follows:

 

, and when importing a jee maven project on Eclipse, the the JBoss Tool plugin returns the following error message:

An internal error occurred during: "Importing Maven projects".

class org.eclipse.jdt.internal.core.JavaNature cannot be cast to class org.eclipse.jdt.core.IJavaProject (org.eclipse.jdt.internal.core.JavaNature and org.eclipse.jdt.core.IJavaProject are in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @21fd7397)

>SOLUTION

After checking if the OpenJDK 17 is not corrupted, the issue was solved by adding some additional JBoss plugins, pointing to menu, Help, Install New Software.

Fill the "Work with:" input box with:

JBoss Tools - Core + Central Update Site - TOOLS-ssh/photon/stable/composite/4.29.1/ - https://download.jboss.org/jbosstools/photon/stable/composite/4.29.1/