I'm following this online course by IBM: Advanced Machine Learning course.
I would like to load a .csv file and then store it as a parquet format in my Cloud Object Storage inside my Watson Project (here you can find how to access to the storage Object Storage Watson).
The teacher of the course is using the following method: add files to Object Storage.
Basically the documentation shows how to automatically retrieve the Object Cloud credential and URL using Insert SparkSession Dataframe option.
I can't find this option at all, i tried almost everything, but clicking on my imported .csv (as suggested in the documentation) only gives me pandas Dataframe option.
What am i doing wrong? I followed each step carefully, i tried almost all the possible envirorments available (Free CPU Python 3.6, Apache Spark Python 3.6 ecc...) with no success, the option won't show at all.
I solved the problem.
You need to create a new envirorment and add Spark to it.
Then once you create your notebook, select this envirorment, load a dummy .json file and you'll be able to select Insert SparkSession Dataframe.