rdatabrickssparkrsparklyr

How to access a Databricks database with sparklyr

New in Azure Databricks environment, I discover the packages SparkR and sparklyr.

From my notebooks with SparkR, I manage to connect to a database :

library(SparkR)
DW <- sql("select * from mydb.sometable")

It runs well, but SparkR syntax seems too far removed from classic R syntax (according to me).

So I wanted to try sparklyr, but I can't access this same database :

library(sparklyr)
sc <- spark_connect(method="databricks")
test <- spark_read_table(sc, "mydb.sometable")

Error : org.apache.spark.sql.AnalysisException: It is not allowed to add database prefix ...

Is something wrong ?

Thanks !!

Solution

  • If it helps anyone, here's what I found that seems to work.

    1. Setting the default data base
    2. Read table in default data base
    library(sparklyr)
library(dplyr)
sc <- spark_connect(method="databricks")
tbl_change_db(sc, "mydb")
foo <- spark_read_table(sc,"sometable")