javaspring-bootdownloadwebclientwebflux

How to combine a WebFlux WebClient DataBuffer download with more actions


I am trying to download a file (or multiple files), based on the result of a previous webrequest. After downloading the file I need to send the previous Mono result (dossier and obj) and the file to another system. So far I have been working with flatMaps and Monos. But when reading large files, I cannot use the Mono during the file download, as the buffer is too small.

Simplified the code looks something like this:

var filePath = Paths.get("test.pdf");
this.dmsService.search()
    .flatMap(result -> {
        var dossier = result.getObjects().get(0).getProperties();
        var objectId = dossier.getReferencedObjectId();
        return Mono.zip(this.dmsService.getById(objectId), Mono.just(dossier));
    })
    .flatMap(tuple -> {
        var obj = tuple.getT1();
        var dossier = tuple.getT2();

        var media = this.dmsService.getDocument(objectId);
        var writeMono = DataBufferUtils.write(media, filePath);
        return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);
    })
    .flatMap(tuple -> {
        var obj = tuple.getT1();
        var dossier = tuple.getT2();
        var objectId = dossier.getReferencedObjectId();
    
        var zip = zipService.createZip(objectId, obj, dossier);
        return zipService.uploadZip(Flux.just(zip));
    })
    .flatMap(newWorkItemId -> {
        return updateMetadata(newWorkItemId);
    })
    .subscribe(() -> {
        finishItem();
    });

dmsService.search(), this.dmsService.getById(objectId), zipService.uploadZip() all return Mono of a specific type. dmsService.getDocument(objectId) returns a Flux due to support for large files. With a DataBuffer Mono it was worked for small files if I simply used a Files.copy:

    ...
    var contentMono = this.dmsService.getDocument(objectId);
    return contentMono;
})
.flatMap(content -> {
    Files.copy(content.asInputStream(), Path.of("test.pdf"));
    ...
}

I have tried different approaches but always ran into problems.

Based on https://www.amitph.com/spring-webclient-large-file-download/#Downloading_a_Large_File_with_WebClient

DataBufferUtils.write(dataBuffer, destination).share().block();

When I try this, nothing after .block() is ever executed. No download is made.

Without the .share() I get an exception, that I may not use block: java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-5

Since DataBufferUtils.write returns a Mono my next assumption was, that instead of calling block, I can Mono.zip() this together with my other values, but this never returns either.

var media = this.dmsService.getDocument(objectId);
var writeMono = DataBufferUtils.write(media, filePath);
return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);

Any inputs on how to achieve this are greatly appreachiated.


Solution

  • I finally figured out that if I use a WritableByteChannel which returns a Flux<DataBuffer> instead of a Mono<Void> I can map the return value to release the DataBufferUtils, which seems to do the trick. I found the inspiration for this solution here: DataBuffer doesn't write to file

    var media = this.dmsService.getDocument(objectId);
    var file = Files.createTempFile(objectId, ".tmp");
    WritableByteChannel filechannel = Files.newByteChannel(file, StandardOpenOption.WRITE);
    var writeMono = DataBufferUtils.write(media, filechannel)
        .map(DataBufferUtils::release)
        .then(Mono.just(file));
    return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);