rfreadread.csv

Website forbidden when scraping web data in R but works fine in browser

I'm trying to import the data here:

https://download.bls.gov/pub/time.series/cu/cu.series

But when I run

fread('https://download.bls.gov/pub/time.series/cu/cu.series')

I get:

Error in curl::curl_download(input, tmpFile, mode = "wb", quiet = !showProgress) : 
  HTTP error 403.

Update: Still getting error even when using custom user agent in Rstudio Cloud enter image description here

Solution

  • This worked for me at the BLS time series site: preface the url with "https://", and then for the user_agent string, insert your email address instead of the browser string. Ex:

    GET("https://download.bls.gov/...",user_agent("youremail@domain.name"))