As the picture below, what's the difference between duration, batch duration and processing time in spark UI ?
thanks
The batch duration of 1 min tells you, that your Spark streaming application works in batches of 1 minute, meaning it plans an RDD every minute. You set this duration in your code when creating the SparkSession.
The processing time tells you that it took Spark 34 seconds to process all input data (provided as input data size).
The duration gives you an understanding of the time it took to finish a particular job within your application.
