rdockerrseleniumdoparallelselenium-remotedriver

Running RSelenium in parallel using Docker

I am currently trying to use the package doParallel in order to parallelise my RSelenium web scraper (running on Docker). I have found this post (Speed up web scraping using multiplie Rselenium browsers) and am copying the answer provided by @hdharrison in here:

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

# using  docker run -d -p 4445:4444 selenium/standalone-chrome:3.5.3
# in windows
URLsPar <- c("https://stackoverflow.com/", "https://github.com/", 
             "http://www.bbc.com/", "http://www.google.com", 
             "https://www.r-project.org/", "https://cran.r-project.org",
             "https://twitter.com/", "https://www.facebook.com/")

appHTML <- c()

(cl <- (detectCores() - 1) %>%  makeCluster) %>% registerDoParallel
# open a remoteDriver for each node on the cluster
clusterEvalQ(cl, {
  library(RSelenium)
  remDr <- remoteDriver(remoteServerAddr = "192.168.99.100", port = 4445L, 
                        browserName = "chrome")
  remDr$open()
})
ws <- foreach(x = 1:length(URLsPar), 
              .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
                print(URLsPar[x])
                remDr$navigate(URLsPar[x])
                remDr$getTitle()[[1]]
              }
> ws
[[1]]
[1] "Stack Overflow - Where Developers Learn, Share, & Build Careers"

[[2]]
[1] "The world's leading software development platform · GitHub"

[[3]]
[1] "BBC - Homepage"

[[4]]
[1] "Google"

[[5]]
[1] "R: The R Project for Statistical Computing"

[[6]]
[1] "The Comprehensive R Archive Network"

[[7]]
[1] "Twitter. It's what's happening."

[[8]]
[1] "Facebook - Log In or Sign Up"     


# close browser on each node
clusterEvalQ(cl, {
  remDr$close()
})

stopImplicitCluster()

This seems to be the solution I am looking for, but when I run it, I come across this error message:

Error in checkForRemoteErrors(lapply(cl, recvResult)) :

3 nodes produced errors; first error: Undefined error in httr call. httr output: Failed to connect to 192.168.99.100 port 4445: Connection refused

This is the 'docker ps' output:

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                    NAMES
f2d62f6b293b        selenium/standalone-chrome:3.5.3   "/opt/bin/entry_poin…"   36 minutes ago      Up 35 minutes       0.0.0.0:4445->4444/tcp   recursing_austin

I understand that I will have to open a new browser for each core, but I assume that's where the issue is: The moment I reduce my cores, I get fewer produced errors.

If I can provide any more details, please let me know! Thanks in advance!

Solution

  • In the mean time, I was able to figure out how to fix my mistake. In case anyone else is facing the same issue, I am leaving a comment here. I cannot explain the logic behind it, but my code runs as expected when I replace 

    remDr <- remoteDriver(remoteServerAddr = "192.168.99.100", port = 4445L, 
                       browserName = "chrome")

    by 

    remDr <- remoteDriver(port = 4445L)

    and using a Firefox browser rather than Chrome.