My app implements an image download function using NodeJS https module. Downloading is fine, but sometimes (no clear cause), the downloaded image were incomplete / contains gray area
const fs = require('fs'), https = require('https');
// ...
return new Promise(async (res) => {
// ...
// assume var url = 'https://example.com/image.jpg'
let file = fs.createWriteStream('./image.jpg');
await https.get(url, async (response) => {
response.pipe(file).on("finish", () => res(true))
}).on("error", err => res(err.stack))
})
as you can see above (and per description of the image), the failed image didn't throw any error ( .on('err'... failed to catch anything)
how can I check / validate that the downloaded image is 100% complete, before I can call the Promise's resolve() ?
I think I found how. So to any of you who ended up with the same question as mine, I'd like to share my answer here
I had to switch to node-fetch, so if you're looking for http solution, then I'm sorry but I don't know how.
const fs = require('fs'), fetch= require('https');
var w_size: number = 0 // file size from URL
await (async (url: string) => {
return new Promise<Buffer>(resolve =>
resolve(fetch(url)
.then((r) => { /* get file size from header */ w_size = parseInt(r.headers.get('content-length') || '0'); return r })
.then(res => res.arrayBuffer())
.then(buff => Buffer.from(buff)))
)
})(url).then(async (buffer) => {
console.log('Writing');
await fs.writeFileSync(downloadedFilePath, buffer);
if (w_size) { // if w_size == 0 (means failed to get ['content-length'] from headers), jump to else
var getPercent = fs.statSync(downloadedFilePath).size;
if (Math.round((getPercent / w_size) * 100) < 99) { // minimum of 99 percent complete, or if you want to be strict, use <= 99 to make sure it's 100%
// Partial download (download failed)
console.log(`Error:\nBuffer end before whole file finished downloading (partial download)`)
throw new Error('Partial download detected')
}
else {...} // Download success
}
else {...} // default behavior if failed to get w_size from URL (means original / meant-to-be size is unknown)
})