scalahttpstreamingscala-dispatch

How to stream the response of dispatcher?


I have to export the CSV data. The volume of data is very high. So I am streaming the response from the microservice. We hit our microservice using dispatcher.

def stream(method: String, urlString: String): Future[Source[ByteString, NotUsed]] =
    method match {
      case GET    =>
        val request = Http(url(urlString))
        request.map { response =>
          response.getStatusCode match {
            case StatusOk => Source.single(ByteString(response.getResponseBody))
          }
        }
    }

It will bring all the data. So to fix this issue, I like to modify it and streamed the data from here as well.

I searched a lot and found this question Scala dispatch stream response line by line

But it has no answer.

Thanks and any help will be appreciated.


Solution

  • After searching much, I read it as Input stream and converted to Akka Stream. It worked for me.

    def stream(method: String, urlString: String): Future[Source[ByteString, Future[IOResult]]] =
        method match {
          case GET    =>
            val futureStream = Http(url(urlString) > as.Response(_.getResponseBodyAsStream))
            futureStream.map { inputStream =>
              val source = () => inputStream
              StreamConverters.fromInputStream(source)
            }
        }