listscalaasynchronousfutureserial-processing

Calling arbitrary number of WS.url().get() in sequence


I have a List[String] of URLs that I want to load and process (parse, store to database) in sequence.

I found only fixed-length examples, like:

    def readUrls = Action {
    implicit request => {
        implicit val context = scala.concurrent.ExecutionContext.Implicits.global

        val url1 = "http://some-website.com"
        val url2 = "http://other-website.com"

        Async {
            for {
                result1 <- WS.url(url1).get()
                result2 <- WS.url(url2).get()
            } yield {
                Ok(result1.body + result2.body)
            }
        }
}

But instead of url1 and url2, I need to process this puppy:

val urls = List("http://some-website.com", "http://other-website.com")

Thanks a bunch for any tips and advice!


Solution

  • If you want to chain Futures together arbitrarily in sequence, foldLeft ought to do the job:

    urls.foldLeft(Future.successful[String]("")){ case (left, nextUrl) =>
        left.flatMap{ aggregatedResult =>
            WS.url(nextUrl).get().map( newResult =>
                aggregatedResult + newResult.body
            )
        }
    }
    

    Since you're just combining the request bodies together, I gave the foldLeft an initial value of a Future empty String, which each step in the fold will then add on the next response body.