javascriptnode.jsdownloaddata-transferchunking

How to download a big file directly to the disk, without storing it in RAM of a server and browser?


I want to implement a big file downloading (approx. 10-1024 Mb) from the same server (without external cloud file storage, aka on-premises) where my app runs using Node.js and Express.js.

I figured out how to do that by converting the entire file into Blob, transferring it over the network, and then generating a download link with window.URL.createObjectURL(…) for the Blob. Such approach perfectly works as long as the files are small, otherwise it is impossible to keep the entire Blob in the RAM of neither server, nor client.

I've tried to implement several other approaches with File API and AJAX, but it looks like Chrome loads the entire file into RAM and only then dumps it to the disk. Again, it might be OK for small files, but for big ones it's not an option.

My last attempt was to send a basic Get-request:

const aTag = document.createElement("a");
aTag.href = `/downloadDocument?fileUUID=${fileName}`;
aTag.download = fileName;
aTag.click();

On the server-side:

app.mjs

app.get("/downloadDocument", async (req, res) => {

    req.headers.range = "bytes=0";

    const [urlPrefix, fileUUID] = req.url.split("/downloadDocument?fileUUID=");

    const downloadResult = await StorageDriver.fileDownload(fileUUID, req, res);

});

StorageDriver.mjs

export const fileDownload = async function fileDownload(fileUUID, req, res) {

    //e.g. C:\Users\User\Projects\POC\assets\wanted_file.pdf
    const assetsPath = _resolveAbsoluteAssetsPath(fileUUID);

    const options = {
        dotfiles: "deny",
        headers: {
            "Content-Disposition": "form-data; name=\"files\"",
            "Content-Type": "application/pdf",
            "x-sent": true,
            "x-timestamp": Date.now()
        }
    };

    res.sendFile(assetsPath, options, (err) => {

        if (err) {
            console.log(err);
        } else {
            console.log("Sent");
        }

    });

};

When I click on the link, Chrome shows the file in Downloads but with a status Failed - No file. No file appears in the download destination.

My questions:

  1. Why in case of sending a Get-request I get Failed - No file?

  2. As far as I understand, res.sendFile can be a right choice for small files, but for big-ones it's better to use res.write, which can be split into chunks. Is it possible to use res.write with Get-request?

P.S. I've elaborated this question to make it more narrow and clear. Previously this question was focused on downloading a big file from Dropbox without storing it in the RAM, the answer can be found: How to download a big file from Dropbox with Node.js?


Solution

  • Chrome can't display the progress of a download effectively because the file is being downloaded in the background. After the download is complete, a link to the file is created and "clicked" to prompt Chrome to display the dialog for the already downloaded file.

    This can be done more easily. You need to create a GET request and let the browser download the file.

    app.get("/download", async (req, res, next) => {
      const { fileName } = req.query;
      const downloadResult = await StorageDriver.fileDownload(fileName);
      res.set('Content-Type', 'application/pdf');
      res.send(downloadResult.fileBinary);
    });
    
    function fileDownload(fileName) {
      const a = document.createElement("a");
      a.href = `/download?fileName=${fileName}`;
      a.download = fileName;
      a.click();
    }