I am trying to get the FileSystem object so that I can use the files from cloud storage without using the dbutils utility functionality. This would allow my code to run on any platform along with databricks. I am using the Databricks Unity catalog enabled workspace and when I try to run the below line of code it gives me error of sparkContext not found.
Code:
import org.apache.hadoop.fs.{FileSystem, FileUtil, Path}
val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
Error text:
cmd4.sc:2: value sparkContext is not a member of org.apache.spark.sql.SparkSession
val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
^Compilation Failed
I have also tried just running spark.sparkContext and it gives the same error. I am running this on databricks cluster with DBR version 13.3 LTS and Access Mode as Shared.
I don't know or am not able to figure out whether this is a unity catalog related issue or something else because earlier I was able to run this code on the databricks cluster without unity catalog tag associated with the cluster.
Looking for any idea/suggestions/solutions that can help me solve this error.
I encountered a similar issue and resolved it by changing the cluster's Access Mode to "Single User" in the configuration.
The "Shared" option comes with limited functionalities, particularly with the Scala preview, which may lead to the error you're experiencing.
Cluster Configuration - Access Mode
I hope this helps you resolve your issue!
