scalascala-dispatchasynchttpclient

Decode a streaming GZIP response in Scala Dispatch?


Receiving a Gzipped response from an API, but Dispatch 0.9.5 doesn't appear to have any methods to decode the response. Any ideas?

Here's my current implementation, the println only prints out string representations of bytes.

   Http(
      host("stream.gnip.com")
      .secure
      .addHeader("Accept-Encoding", "gzip")
       / gnipUrl
      > as.stream.Lines(println))()

Tried to look at implementing my own handler, but not sure where to begin. Here's the relevant file for Lines: https://github.com/dispatch/reboot/blob/master/core/src/main/scala/as/stream/lines.scala

Thanks!


Solution

  • Simply abandoned Dispatch and used Java APIs directly. Disappointing, but it got the job done.

      val GNIP_URL = isDev match {
        case true => "https://url/apath/track/dev.json"
        case false => "https://url/path/track/prod.json"
      }
      val GNIP_CHARSET = "UTF-8"
    
      override def preStart() = {
        log.info("[tracker] Starting new Twitter PowerTrack connection to %s" format GNIP_URL)
    
        val connection = getConnection(GNIP_URL, GNIP_USER, GNIP_PASSWORD)
        val inputStream = connection.getInputStream()
        val reader = new BufferedReader(new InputStreamReader(new StreamingGZIPInputStream(inputStream), GNIP_CHARSET))
        var line = reader.readLine()
        while(line != null){
            println(line)
            line = reader.readLine()
        }
      }
    
      private def getConnection(urlString: String, user: String, password: String): HttpURLConnection = {
        val url = new URL(urlString)
    
        val connection = url.openConnection().asInstanceOf[HttpURLConnection]
        connection.setReadTimeout(1000 * 60 * 60)
        connection.setConnectTimeout(1000 * 10)
    
        connection.setRequestProperty("Authorization", createAuthHeader(user, password));
        connection.setRequestProperty("Accept-Encoding", "gzip")
        connection
      }
    
      private def createAuthHeader(username: String, password: String) = {
        val encoder = new BASE64Encoder()
        val authToken = username+":"+password
       "Basic "+encoder.encode(authToken.getBytes())
      }
    

    Used GNIP's example: https://github.com/gnip/support/blob/master/Premium%20Stream%20Connection/Java/StreamingConnection.java