Spring boot REST API Streaming large file download from a Server using WebClient and DataBufferUtils

I'm writing a API from a Rest controller to download large file from a server without overloading the memory. For that, i'm trying to download streaming the file.

For now, i have use WebClient to send the GET request then, write the response on the HttpServletResponse

Here is my API code

@GetMapping(value = "/download/streaming/{fileId}", produces = { "application/pdf", "application/json" })
public void downloadDocumentStreaming(@Parameter(name = "fileId") @PathVariable("fileId") String fileId,
        HttpServletResponse response) {
    try {
        HttpHeaders httpHeaders = //Some http header
        httpHeaders.setAccept(List.of(MediaType.APPLICATION_OCTET_STREAM));

        Flux<DataBuffer> flux = WebClient.create().get()
                .uri(builder -> builder.host(serverHostUrl).port(serverPort).scheme("https")
                        .path("/dowload/file/{fileId}").build(fileId))
                .headers(headers -> headers.addAll(httpHeaders)).retrieve().bodyToFlux(DataBuffer.class);

        DataBufferUtils.write(flux, response.getOutputStream()).share().blockLast();

    } catch (IOException e) {
        e.printStackTrace();
    }
}

To test if my API is not overloading the memory, I run the application with the following VM argument: -Xmx64m

The download is working for file less than 64 MB But for biggest file (example: 74 MB file), I'm getting the following error while testing from Postman.

Caused by: java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 63010883, limit: 67108864)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178) ~[na:na]
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[na:na]
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[na:na]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:139) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:129) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140) ~[netty-buffer-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

It seems the streaming is not really working as the memory is getting overloaded. In the other hand, I have another issue with swagger Ui. Even for small file size, swagger Ui is not able to render the response and is throwing the error Unrecognized response type; displaying content as text.

I will really be happy if someone can have a workaround to sole the issue.

Thanks a lot.

I assume you downloaded the large file from non-blocking server. Then we can use StreamingResponseBody to write directly to an OutputStream before passing that written information back to the client using a ResponseEntity. Your method will look like this

@GetMapping(value = "/download/streaming/{fileId}", produces = { "application/pdf", "application/json" })
public ResponseEntity<StreamingResponseBody> downloadDocumentStreaming(@Parameter(name = "fileId") @PathVariable("fileId") String fileId) {

    HttpHeaders httpHeaders = //Some http header
        httpHeaders.setAccept(List.of(MediaType.APPLICATION_OCTET_STREAM));

    Flux<DataBuffer> flux = WebClient.create().get()
            .uri(builder -> builder.host(serverHostUrl).port(serverPort).scheme("https")
                    .path("/dowload/file/{fileId}").build(fileId))
            .headers(headers -> headers.addAll(httpHeaders)).retrieve().bodyToFlux(DataBuffer.class);

    StreamingResponseBody stream = outputStream -> Mono.create(sink ->
            DataBufferUtils.write(flux, outputStream).subscribe(DataBufferUtils::release,
                sink::error,
                sink::success))
        .block();

    return ResponseEntity.ok()
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename="+"yourfilename.pdf")
        .body(stream);
}

Note that, if you're using Postman for testing then you need to set up the Max response in Postman as well.