How to connect RStudio Cloud to Spark?
I am using RStudio Cloud and I want to connect to Spark using sparklyr package. I tried a local master and a yarn master. The code is as below.
library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "local")
sc <- spark_connect(master = "yarn")
# Error in system2(file.path(spark_home, "bin", "spark-submit"), "--version", : error in running command
Neither worked. I don't know how to set up the Spark environments further. Any help would be much appreciated.
Solution
This could be a problem with the version of Spark.
This works fine for me, on a new project on RStudio Cloud:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "3.0.0")
sc <- spark_connect(master = "local")
- Converting the row of a data.table to a vector
- ggplot multi-line log-log labels
- dplyr filter vector by position in pipe %>%
- Removing white spaces in a histogram?
- How to assign and match land cover type from Corine to a dataframe with a set of lon lat coordinates?
- Error in list2env(list, envir = .GlobalEnv) : names(x) must be a character vector of the same length as x
- How to traverse hclust internal node in R
- How are the abbreviations from the par function like mgp, mar, mai, deciphered?
- How to map over function with multiple arguments (and changing data)?
- Join/fill in data based on both row and column values
- How to plot a Log Pearson III distribution using `ggplot2` in R?
- Variable names and Easystats reports
- R plotly(): Adding regression line to a correlation scatter plot
- How to Use ifelse Statements in R
- Creating subgraphs from neighbors with max weight in R using igraph
- merge based on id and dates with missing data
- Automatically adjust the size of the text inside each bar on a bar plot using ggplot2 in R
- gghighlight and facet_wrap one always highlighted
- Mask and Crop functions in Terra R
- How to use a variable in dplyr::filter?
- Changing R Temporary Files
- How do you add a general label to facets in ggplot2?
- change colour of controls in R DT datatable
- Click Button Within Datatable to Display Popup Twice in a row
- How to display values in filter with ascending order
- Select Data From Column in Data Table
- Warning: Error in dataTableOutput: unused argument (height = "auto")
- How to have cells with rounded corners in {gt}?
- Two colours in the same box
- Change color of datatable filter dropdown box