javascriptflaskfetch-apishutil

Downloading an archive while it is being created with shutil, flask and javascript


I am building this program where I index my external hard drive on a flask web app, where files can be downloaded. Before downloading a folder via flask.send_file() it needs to be archived first because flask can't send directories only files.

So I create the archive via shutil.make_archive() and after it is done the download starts.

The problem here is that some of the files can get quite large and even if shutil can handle them, it can take some time before they are done archiving since this is on an old external hard drive.

Fortunately when shutil starts creating the archive it is visible so that might be of some use.

Here is the javascript that handles the downloading and the creation of the zip file. This works as intended:

async function download_file() {
    var download_path;
    if (!(path.includes('.'))) {
        const promise = await fetch(`/archive?path=${path}&name=${select}`, {method: 'GET'});
        download_path = await promise.json();
    } else {
        download_path = path;
    }
    var downloader_a = document.createElement('a');
    downloader_a.href = `/download?path=${download_path}`;
    downloader_a.click();
}

Accompanied by the python:

@app.route('/download')
def download():
    path = request.args.get('path')
    return send_file(drive + path, as_attachment=True)


@app.route('/archive')
def archive_folder():
    path = drive + request.args.get('path')
    name = request.args.get('name')
    shutil.make_archive('{}{}/{}'.format(drive, archiving_folder, name), 'zip', path)
    return jsonify('{}/{}.zip'.format(archiving_folder, name))

So I believe this problem arises due to the asynchronous nature of my javascript functions. What I have tried was to remove the await from this javascript that creates the archive so that it goes from this:

const promise = await fetch(`/archive?path=${path}&name=${select}`, {method: 'GET'});
download_path = await promise.json();

To this:

const promise = fetch(`/archive?path=${path}&name=${select}`, {method: 'GET'});
download_path = promise.json();

I thought that this might be a long shot becuase what is happening when I do this, is that the incomplete archive gets downloaded, but I want to download the archive as it is being made. Any thougths?

Thanks in advance.


Solution

  • By using zipstream-ng, the resource dave recommended in the comments, I was able to precisely implement what I was looking for.

    After I installed zipstream-ng and went through the example I changed my javascript download function to:

    async function download_file() {
    var downloader_a;
    downloader_a = document.createElement('a');
    if (!(path.includes('.'))) {
        downloader_a.href = `/zip_stream?path=${path}&name=${selected_item}`
    } else {
        downloader_a.href = `/download?path=${path}`;
    }
    downloader_a.click();
    

    With corresponding python:

    from zipstream import ZipStream, ZIP_DEFLATED
    @app.route('/zip_stream')
    def stream_zip_file():
        path = drive + request.args.get('path')
        name = request.args.get('name')
        zs = ZipStream.from_path(path)
        return Response(
            zs,
            mimetype='application/zip',
            headers={
                "Content-Disposition": f"attachment; filename={name}.zip",
                "Content-Length": len(zs),
                "Last-Modified": zs.last_modified,
            }
        )
    

    Now when the folder is being downloaded a zip file of the folder will be streamed.