amazon-web-servicespysparkaws-gluedirected-acyclic-graphsspark-ui

How to view AWS Glue Spark UI


In my Glue job, I have enabled Spark UI and specified all the necessary details (s3 related etc.) needed for Spark UI to work.
How can I view the DAG/Spark UI of my Glue job?


Solution

  • You need to setup an ec2 instance that can host the history server.

    The below documentation has links to CloudFormation templates that you can use. https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html

    You can access the history server via the ec2 instance(default on 18080). You need to configure the networks and ports suitably.

    EDIT - There is also an option to setup SparkUI locally. This requires downloading the docker image from aws-glue-samples repo amd settin the AWS credential and s3 location there. This server consummes the files that the glue job generates. The files are about 4MB large.