[SOLVED] Are TaskMetrics available via stageCompleted SparkListener.onStageCompleted() aggregated metrics for all the tasks in the stage?

Are TaskMetrics available via stageCompleted SparkListener.onStageCompleted() aggregated metrics for all the tasks in the stage?

I'm writing my own SparkListener to send metrics to 3rd party systems. At the end of the stage I'd like to send the aggregated metrics from all the tasks within this stage.

SparkListener has onStageCompleted() method, which I can override to retrieve the TaskMetrics as per the code snippet below:

 @Override
    public void onStageCompleted(SparkListenerStageCompleted stageCompleted) {
        TaskMetrics taskMetrics = stageCompleted.stageInfo().taskMetrics();
    }

Question: What the metrics from the taskMetrics objects above represents? Are these the metrics for most recent task from the stage or maybe aggreagated metrics for all the tasks within the stage?

Docs of the 3.2.3 version I'm using: https://spark.apache.org/docs/3.2.3/api/java/index.html?org/apache/spark/scheduler/SparkListener.html

I checked the documentation, but couldn't find any information on this topic.

Solution

As I was real hoping for a quick answer, just after posting the message I started experimenting on my own. I've tried a few Executor Task Metrics like executorRunTime, inputMetrics.bytesRead, shuffleWriteMetrics.bytesWritten and retrieved the metrics on both end of the task & end of the stage:

@Override
public void onTaskEnd(SparkListenerTaskEnd taskEnd) {
        TaskMetrics taskMetrics = taskEnd.taskMetrics();
}

@Override
public void onStageCompleted(SparkListenerStageCompleted stageCompleted) {
        TaskMetrics taskMetrics = stageCompleted.stageInfo().taskMetrics();
}

After comparing the values I confirm that the stage-level metrics are the sum of the metrics from individual tasks.