rhttpcurlshinyapps

HTTP issue with curl on shinyapps.io

I need your help!

Issue

I have an R Shiny app hosted on shinyapps.io on which I'm recently having issues downloading a 10 MB file from the internet. When I run the app locally, everything still runs perfectly.

Analysis

Since the issue appeared, I have spotted the following in the shinyapps.io logs:

Warning: Error in open.connection: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

One solution would be to force HTTP/1.1 as indicated here and here. I have tried it but I stumble on another issue. The file is still not downloaded and the logs indicate the following:

old SSL session ID is stale, removing

Example

I have prepared a full example here on shinyapps.io and below for you to reproduce:

library(shiny)
library(curl)
library(readr)

ui <- fluidPage(
    titlePanel("Debug HTTP from shinyapps.io to edf.fr"),
    sidebarLayout(
      sidebarPanel(
        radioButtons(
          "radio", "Select option",
          choices = list("HTTP/2" = 1, "HTTP/1.1" = 2), selected = 1
        )
      ),
      mainPanel(textOutput("text"))
    )
)

server <- function(input, output) {
  url <- "https://www.edf.fr/doaat/export/light/csv"
  h <- new_handle(failonerror = TRUE, verbose = TRUE)
  output$text <- renderText({
    if (input$radio == 2) {
      handle_setopt(h, http_version = 2)
    }
    req <- curl_fetch_memory(url, handle = h)
    file <- rawToChar(req$content)
    read_lines(file, n_max=1, locale=locale(encoding='latin1')) })
}

shinyApp(ui = ui, server = server)

The example above works perfectly when it is run locally (i.e. success = the first line of the downloaded 10 MB CSV file appears on the main panel) but fails both in HTTP/2 and HTTP/1.1 when running on shinyapps.io.

It seems this has to do with the curl version on shinyapps.io and/or faulty HTTP transactions with the servers (shinyapps.io or edf.fr). So I share more verbose curl logs below for the HTTP experts here.

HTTP/2 logs

Listening on http://127.0.0.1:37633
*   Trying 104.68.246.156:443...
* Connected to www.edf.fr (104.68.246.156) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=FR; ST=�le-de-France; L=Paris; O=ELECTRICITE DE FRANCE SA; CN=edf.fr
*  start date: Mar 22 00:00:00 2024 GMT
*  expire date: Mar 21 23:59:59 2025 GMT
*  subjectAltName: host "www.edf.fr" matched cert's "www.edf.fr"
*  issuer: C=IE; O=DigiCert Ireland Limited; CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5c8f626765d0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /doaat/export/light/csv HTTP/2    
Host: www.edf.fr    
user-agent: R/4.4.1 (ubuntu-22.04) R (4.4.1 x86_64-pc-linux-gnu x86_64 linux-gnu)   
accept: */* 
accept-encoding: deflate, gzip, br, zstd    
    
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
* stopped the pause stream!
* Connection #0 to host www.edf.fr left intact
Warning: Error in curl_fetch_memory: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

HTTP/1.1 logs

Listening on http://127.0.0.1:36691
*   Trying 104.68.246.156:443...
* Connected to www.edf.fr (104.68.246.156) port 443 (#0)
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=FR; ST=�le-de-France; L=Paris; O=ELECTRICITE DE FRANCE SA; CN=edf.fr
*  start date: Mar 22 00:00:00 2024 GMT
*  expire date: Mar 21 23:59:59 2025 GMT
*  subjectAltName: host "www.edf.fr" matched cert's "www.edf.fr"
*  issuer: C=IE; O=DigiCert Ireland Limited; CN=DigiCert G2 TLS EU RSA4096 SHA384 2022 CA1
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /doaat/export/light/csv HTTP/1.1  
Host: www.edf.fr    
User-Agent: R/4.4.1 (ubuntu-22.04) R (4.4.1 x86_64-pc-linux-gnu x86_64 linux-gnu)   
Accept: */* 
Accept-Encoding: deflate, gzip, br, zstd    
    
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
Solution

  • As a workaround, perhaps make that request with httr2, req_perform(..., path = dest) should store content even if there are some issues with closing that connection.

    In following example resp$body points to a tmp file where content is stored.

    library(shiny)
library(httr2)
library(readr)

ui <- fluidPage(
  titlePanel("Debug HTTP from shinyapps.io to edf.fr"),
  mainPanel(
    verbatimTextOutput("resp"),
    verbatimTextOutput("text"))
  )

server <- function(input, output) {
  url <- "https://www.edf.fr/doaat/export/light/csv"
  tmp_csv <- tempfile()
  resp <- 
    request(url) |> 
    req_perform(path = tmp_csv)
  output$resp <- renderPrint(str(resp))
  output$text <- renderPrint(read_lines(resp$body, n_max=1, locale=locale(encoding='latin1')))
}

shinyApp(ui = ui, server = server, options = list(display.mode = "showcase"))

    Currently deployed as https://margusl.shinyapps.io/shiny_so240906/