How to use scala sttp client to download file

I have task to check AWS pricing, it's a public json file. there is no auth needed.

I use Tapir Sttp library

    val request: Request[Either[String, String], Any] = basicRequest
        .header(HeaderNames.Accept, MediaType.ApplicationOctetStream.toString())
        .get(uri"https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json")


    val response = request.send(sttpBackend)
    val res = Await.result(response, 30.seconds)

above is my simple get call. But I got error Caused by: java.io.UnsupportedEncodingException: Unsupported encoding

I can download the file from browers, get the json from Postman. Anybody know how to download file by sttp client Thanks

try to download json file from amazon pricing endpoint

Solution

It seems the aws endpoint returns a response with an empty Content-Encoding header:

> GET /offers/v1.0/aws/index.json HTTP/1.1
> Host: pricing.us-east-1.amazonaws.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
(...)
< x-amz-server-side-encryption: AES256
< Content-Encoding:
< x-amz-version-id: RPn8P8KKFZDsbbPZSHSPEAxgOPJ4okmH
(...)

Our code tries to match one of the supported decoding algorithms to the value of the header, if it is present. This fails, as there is no algorithm for an empty string.

I've created an issue to fix that in sttp. The fix is to ignore empty values of the header.

There is also a work-around: you can provide a custom encoding handler to "handle" the empty value. For example, using the HttpURLConnectionBackend:

object Test extends App {
  val request: Request[Either[String, String], Any] = basicRequest
    .header(HeaderNames.Accept, MediaType.ApplicationOctetStream.toString())
    .get(uri"https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json")

  val backend = HttpURLConnectionBackend(customEncodingHandler = { case (is, "") => is })

  println(request.send(backend))
}