node.jsimagedownloadimagedownload

Node JS https check if downloaded image is 100% complete


My app implements an image download function using NodeJS https module. Downloading is fine, but sometimes (no clear cause), the downloaded image were incomplete / contains gray area

const fs = require('fs'), https = require('https');
// ...
        return new Promise(async (res) => {
            // ...
            // assume var url = 'https://example.com/image.jpg'
            let file = fs.createWriteStream('./image.jpg');
            await https.get(url, async (response) => {
                response.pipe(file).on("finish", () => res(true))
            }).on("error", err => res(err.stack))
        })

2 example: 100%, and failed

as you can see above (and per description of the image), the failed image didn't throw any error ( .on('err'... failed to catch anything)

how can I check / validate that the downloaded image is 100% complete, before I can call the Promise's resolve() ?


Solution

  • I think I found how. So to any of you who ended up with the same question as mine, I'd like to share my answer here

    I had to switch to node-fetch, so if you're looking for http solution, then I'm sorry but I don't know how.

    const fs = require('fs'), fetch= require('https');
    var w_size: number = 0 // file size from URL
    
    await (async (url: string) => {
        return new Promise<Buffer>(resolve =>
            resolve(fetch(url)
                .then((r) => { /* get file size from header */ w_size = parseInt(r.headers.get('content-length') || '0'); return r })
                .then(res => res.arrayBuffer())
                .then(buff => Buffer.from(buff)))
        )
    })(url).then(async (buffer) => {
        console.log('Writing');
        await fs.writeFileSync(downloadedFilePath, buffer);
        if (w_size) { // if w_size == 0 (means failed to get ['content-length'] from headers), jump to else
            var getPercent = fs.statSync(downloadedFilePath).size;
            if (Math.round((getPercent / w_size) * 100) < 99) { // minimum of 99 percent complete, or if you want to be strict, use <= 99 to make sure it's 100%
                // Partial download (download failed)
                console.log(`Error:\nBuffer end before whole file finished downloading (partial download)`)
                throw new Error('Partial download detected')
            }
            else {...} // Download success
        }
        else {...} // default behavior if failed to get w_size from URL (means original / meant-to-be size is unknown)
    })