apache-sparkramspark-shell

How to release RAM memory after stopping a Spark-Shell instance?

I've observed that the RAM holds on to data even after stopping/quitting the spark-shell instance. How can I free RAM memory?

Solution

  • Even after you quit your spark-shell instance, the spark application will still be running. For the RAM to be released, you can try killing the application instance either using the web UI end point or directly on your shell

    You can find the application process by the following

    ps -aef|grep spark-shell

    Once you find the process, then you can kill it by

    kill -9 pid