scalaamazon-s3akkaalpakka

Alpakka and S3 truncating downloaded files


I have a simple piece of code, based on alpakka examples, which should download some file from S3 for further processing:

S3.download(bucket, file)
  .runWith(Sink.head)
  .flatMap {
    case Some((data, _)) =>
      data.map(_.utf8String).runWith(Sink.head).map(Some.apply)
    case None =>
      Future.successful(None)
    }

The problem is that the file content is getting truncated, the file size from the ObjectMetadata is correct it has ~2M it isn't a huge file.

What I noticed is, when I use Sink.head the file content is from the beginning to middle if I change it to Sink.last it is from the middle to the end. Am I getting chunks from the file but why are they not seem to be streamed?

Can't figure out what's happening and how to fix this. I believe the issue is the same as this other question, unfortunately without answers.

Thanks


Solution

  • I've found the solution, in the end it was very clear...

    Just need to replace: data.map(_.utf8String).runWith(Sink.head).map(Some.apply)

    with: data.map(_.utf8String).runWith(Sink.seq).map(_.mkString).map(Some.apply)

    accumullating all the chunks from the file.

    Thanks