Access data on EMR directory from EMR Studio: Workspaces (Notebooks)

I have some data saved on s3, which I want to import while running a python script on EMR.

To do it through a python code on EMR console: I just create the directories/file on my EMR like this /home/mysource/settings by copying the files from S3 to EMR and then the following code works as it should.

import sys
from mysource.settings import *

Now I want to do the same thing from EMR studio: Workspaces, but apparently, even after attaching the EMR cluster to a workspace notebook, I am not able to make the import work. It would be even better if I can import directly from s3, without creating this directory/file structure on EMR in first place.

I use an access key and a secret access key to transfer data between s3 and EMR (without tokens).

TLDR:

• When you connect to an EMR cluster from EMR Studio (really a hosted Jupyter notebook), the files in your 'local' file system are not directly accessible to the Spark driver. You should instead write code to save your Python script on the 'local' disk (e.g.: import module from s3 in sagemaker ) and then import it in Python from that path.

More details:

• AWS tries to abstract this from you, but when you connect to an EMR cluster from EMR Studio/SageMaker Studio, it uses Apache Livy (see e.g. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-spark-optimization.html ). The sequence from your Web browser to Spark session is like so:
  • You connect to EMR Studio, a hosted Jupyter notebook backed by a PySpark kernel
  • This kernel in turn connects to the EMR cluster via Apache Livy, a REST server
  • Apache Livy runs on the EMR leader node and creates a Spark Session
• When you try to execute a code cell in your notebook, the code is sent by PySpark kernel to the Apache Livy app running on EMR leader node, an EC2 instance
• But the files on your EMR studio are attached to the kernel, a different EC2 instance that is not part of the EMR cluster, and acts as a pass-through/Apache Livy client and simply passes your code cells to Apache Livy server on EMR leader node and then waits for the output
• Hence the instance on which your PySpark code executes, the EMR leader node, does not see any of your 'EMR Studio files', because they are attached to a different EC2 instance, the PySpark kernel for the Jupyter webserver
• Therefore if you have a Python script you want to import, you have to write code to copy it from S3 on the EMR leader node's file system, and then import it from that location - that's where the Python process executing your code actually runs
• If you are doing this often, you can also automate this via a Bootstrap action, or in our team, we package all our Python libraries into a Docker image, and then use EMR's capability to run Spark driver/executors inside the Docker image: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-docker.html (but I would not do this for quick exploration/one-off set up).