[SOLVED] Getting 'Undefined error in httr call' when using remDr$navigate with R Selenium

Getting 'Undefined error in httr call' when using remDr$navigate with R Selenium - how to fix it?

remDr$navigate (url) error: Undefined error in httr call. httr output: length(url) == 1 is not TRUE

This happens for URL url <- "https://eprel.ec.europa.eu/screen/product/airconditioners" Remote driver is configured without errors. What could be the cause and how to solve it? Is it perhaps that this website is not scrape-able?

I tried checking:

the specification of URL (it's correct)
using get() and navigateTo functions.

My system is 64-bit, and so is R, but the only available versions for webdriver for Chrome and Firefox are 32-bit. Could that be the problem?

The ChatGPT says it could be related to the RSelenium setup or compatibility with the specific website. Could it be that for this website, another scraping tool/package should be used?

Solution

Try this instead:

library(tidyverse)
library(httr2)

"https://eprel.ec.europa.eu/api/products/airconditioners?_page=1&_limit=25&indoorSoundPowerCoolingMin=1&indoorSoundPowerCoolingMax=99&outdoorSoundPowerCoolingMin=0&outdoorSoundPowerCoolingMax=99&indoorSoundPowerHeatingMin=1&indoorSoundPowerHeatingMax=99&outdoorSoundPowerHeatingMin=0&outdoorSoundPowerHeatingMax=99&coolingDesignLoadMin=0.1&coolingDesignLoadMax=99.9&heatingDesignLoadMin=0.1&heatingDesignLoadMax=99.9&sort0=onMarketStartDateTS&order0=DESC&sort1=energyClass&order1=DESC" %>%
  request() %>%
  req_perform() %>%
  resp_body_json(simplifyVector = TRUE, check_type = FALSE) %>%
  pluck("hits")