I'm working on an application with Angular 12, which allows users to store a heavy files, like images, and videos. (There are videos that could be larger than 1GB). Anyway, they are very heavy files.
In this application, it's necessary to place a button "Download everything as ZIP" the problem is that the way I currently have to handle this download is with JSZip and it makes my computer very slow, also it does not report the progress until the file is armed zip, that is, it spends 20 minutes at 0% download and even later it begins to report the progress.
This is the solution that I am currently implementing:
<button (click)="downloadAll()">Download as Zip</button>
Then in my TS file I implement the function
downloadAll() {
// This is the function where I get the access links of all the uploaded files, which can weigh more than 11GB in total.
// But here I only get an array of links
const urls = this.PROJECT.resources.map(u => u.link);
this.downloadService.downloadAll(urls, this.downloadCallback); // my download service
}
// this is my callback function that I send to the service
downloadCallback(metaData) {
const percent = metaData.percent;
setTimeout(() => {
console.log(percent);
if (percent >= 100) { console.log('Zip Downloaded'); }
}, 10);
}
My Download service has the function Download All, where I transform all recollected files in a zip.
downloadAll(urls: string[], callback?: any) {
let count = 0;
const zip = new JSZip();
urls.forEach(u => {
// for each link I undestand that I need to get the Binary Content
const filename = u.split('/')[u.split('/').length - 1];
// I think that this function where the binary content of a file is obtained through a link is what causes the download to be so slow and to consume a lot of resources
JSZipUtils.getBinaryContent(u, (err, data) => {
if (err) { throw err; }
zip.file(filename, data, { binary: true });
count++;
if (count === urls.length) {
// This function works relatively normal and reports progress as expected.
zip.generateAsync({ type: 'blob' }, (value) => callback(value)).then((content) => {
const objectUrl: string = URL.createObjectURL(content);
const link: any = document.createElement('a');
link.download = 'resources.zip';
link.href = objectUrl;
link.click();
});
}
});
});
}
So basically my question is: Is there a correct way to implement downloading multiple large files in a zip?
At all times it is necessary for the user to know the progress of their download, but as I repeat, when obtaining the binary content of each file, it takes a long time to do this and until a long time after the generated file begins to download.
What is the correct way to download large files inside a zip?
So this took my attention went ahead and tried to build it with observables.
short answer: it's not possible to zip big files in browser with Jszip and I feel also it should be done in Backend.
https://stuk.github.io/jszip/documentation/limitations.html
But here is a correct implementation for zipping if needed:
The project I built contains a backend that stream files and the frontend which will take files and zip them with the progress bar.
I tried to zip two movie files with a total of 2.4G and it failed when reached 100 percent error was:
Uncaught (in promise): RangeError: Array buffer allocation failed
I wrapped jszip with a observable:
import {
HttpClient,
HttpEvent,
HttpEventType,
HttpHeaderResponse,
HttpProgressEvent,
HttpResponse,
HttpUserEvent
} from '@angular/common/http';
import {Injectable} from '@angular/core';
import {combineLatest, Observable} from 'rxjs';
import {scan} from 'rxjs/operators';
import JSZip from 'jszip';
export class DownloadModel {
link: string = '';
fileSize: number = 0;
fileName: string = '';
}
export interface DownloadStatus<T> {
progress: number;
state: 'PENDING' | 'IN_PROGRESS' | 'DONE' | 'SENT';
body: T | null;
httpEvent: HttpEvent<unknown> | null;
downloadModel: DownloadModel;
}
export interface ZipStatus<T> {
progress: number;
state: 'PENDING' | 'DOWNLOADING' | 'DOWNLOADED' | 'ZIPPING' | 'DONE';
body: {
downloadModel: DownloadModel,
downloaded: Blob | null
}[];
zipFile: Blob | null;
httpEvent: HttpEvent<unknown> | null;
}
@Injectable({providedIn: 'root'})
export class DownloadService {
constructor(private httpClient: HttpClient) {
}
public downloadMultiple(downloadLinks: DownloadModel[]): Observable<DownloadStatus<Blob>[]> {
return combineLatest(downloadLinks.map((downloadModel): Observable<DownloadStatus<Blob>> => {
return this.httpClient.get(downloadModel.link, {
reportProgress: true,
observe: 'events',
responseType: 'blob'
}).pipe(
scan((uploadStatus: DownloadStatus<Blob>, httpEvent: HttpEvent<Blob>, index: number): DownloadStatus<Blob> => {
if (this.isHttpResponse(httpEvent)) {
return {
progress: 100,
state: 'DONE',
body: httpEvent.body,
httpEvent,
downloadModel
};
}
if (this.isHttpSent(httpEvent)) {
return {
progress: 0,
state: 'PENDING',
body: null,
httpEvent,
downloadModel
};
}
if (this.isHttpUserEvent(httpEvent)) {
return {
progress: 0,
state: 'PENDING',
body: null,
httpEvent,
downloadModel
};
}
if (this.isHttpHeaderResponse(httpEvent)) {
return {
progress: 0,
state: 'PENDING',
body: null,
httpEvent,
downloadModel
};
}
if (this.isHttpProgressEvent(httpEvent)) {
return {
progress: Math.round((100 * httpEvent.loaded) / downloadModel.fileSize),
state: 'IN_PROGRESS',
httpEvent,
body: null,
downloadModel
};
}
console.log(httpEvent);
throw new Error('unknown HttpEvent');
}, {state: 'PENDING', progress: 0, body: null, httpEvent: null} as DownloadStatus<Blob>));
}
)
);
}
zipMultiple(downloadMultiple: Observable<DownloadStatus<Blob>[]>): Observable<ZipStatus<Blob>> {
return new Observable<ZipStatus<Blob>>(((subscriber) => {
downloadMultiple.pipe(
scan((uploadStatus: ZipStatus<Blob>, httpEvent: DownloadStatus<Blob>[], index: number): ZipStatus<Blob> => {
if (httpEvent.some((x) => x.state === 'SENT')) {
return {
state: 'PENDING',
body: [],
httpEvent: null,
progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
zipFile: null
};
}
if (httpEvent.some((x) => x.state === 'PENDING')) {
return {
state: 'PENDING',
body: [],
httpEvent: null,
progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
zipFile: null
};
}
if (httpEvent.some((x) => x.state === 'IN_PROGRESS')) {
return {
state: 'DOWNLOADING',
body: [],
httpEvent: null,
progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
zipFile: null
};
}
if (httpEvent.every((x) => x.state === 'DONE')) {
return {
state: 'DOWNLOADED',
body: httpEvent.map(x => {
return {
downloadModel: x.downloadModel,
downloaded: x.body
};
}),
httpEvent: null,
progress: 50,
zipFile: null
};
}
throw new Error('ZipStatus<Blob> unhandled switch case');
}, {state: 'PENDING', progress: 0, body: [], httpEvent: null, zipFile: null} as ZipStatus<Blob>)
).subscribe({
next: (zipStatus) => {
if (zipStatus.state !== 'DOWNLOADED') {
subscriber.next(zipStatus);
} else {
this.zip(zipStatus.body.map(x => {
return {
fileData: x.downloaded as Blob,
fileName: x.downloadModel.fileName
};
})).subscribe({
next: (data) => {
// console.log('zipping next');
subscriber.next(data);
},
complete: () => {
// console.log('zipping complete');
subscriber.complete();
},
error: (error) => {
// console.log('zipping error');
}
});
}
},
complete: () => {
// console.log('zip$ source complete: ');
},
error: (error) => {
// console.log('zip$ source error: ', error);
}
});
}));
}
private zip(files: { fileName: string, fileData: Blob }[]): Observable<ZipStatus<Blob>> {
return new Observable((subscriber) => {
const zip = new JSZip();
files.forEach(fileModel => {
zip.file(fileModel.fileName, fileModel.fileData);
});
zip.generateAsync({type: "blob", streamFiles: true}, (metadata) => {
subscriber.next({
state: 'ZIPPING',
body: [],
httpEvent: null,
progress: Math.trunc(metadata.percent / 2) + 50,
zipFile: null
});
}).then(function (content) {
subscriber.next({
state: 'DONE',
body: [],
httpEvent: null,
progress: 100,
zipFile: content
});
subscriber.complete();
});
});
}
private isHttpSent<T>(event: HttpEvent<T>): event is HttpResponse<T> {
return event.type === HttpEventType.Sent;
}
private isHttpResponse<T>(event: HttpEvent<T>): event is HttpResponse<T> {
return event.type === HttpEventType.Response;
}
private isHttpHeaderResponse<T>(event: HttpEvent<T>): event is HttpHeaderResponse {
return event.type === HttpEventType.ResponseHeader;
}
private isHttpUserEvent<T>(event: HttpEvent<T>): event is HttpUserEvent<T> {
return event.type === HttpEventType.User;
}
private isHttpProgressEvent(event: HttpEvent<Blob>): event is HttpProgressEvent {
return (
event.type === HttpEventType.DownloadProgress ||
event.type === HttpEventType.UploadProgress
);
}
}
and you can find the code for downloading multiple file at the same tile from repository link
and here is where it will get called:
export class AppComponent {
constructor(
private http: HttpClient,
private downloadService: DownloadService
) {
}
start() {
console.log('start');
const file1 = new DownloadModel();
file1.link = 'http://localhost:3000?name=1.jpg';
file1.fileSize = 41252062;
file1.fileName = '1.jpg';
const file2 = new DownloadModel();
file2.link = 'http://localhost:3000?name=2.jpg';
file2.fileSize = 39986505;
file2.fileName = '2.jpg';
const download$ = this.downloadService.downloadMultiple([file1, file2]).pipe(tap({
next: (data) => {
// console.log('download$ next: ', data);
},
complete: () => {
// console.log('download$ complete: ');
},
error: (error) => {
// console.log('download$ error: ', error);
}
}));
const zip$ = this.downloadService.zipMultiple(download$);
zip$
.pipe(distinctUntilKeyChanged('progress')).subscribe({
next: (data) => {
console.log('zip$ next: ', data);
if (data.zipFile) {
const downloadAncher = document.createElement("a");
downloadAncher.style.display = "none";
downloadAncher.href = URL.createObjectURL(data.zipFile);
downloadAncher.download = 'images.zip';
downloadAncher.click();
}
},
complete: () => {
console.log('zip$ complete: ');
},
error: (error) => {
console.log('zip$ error: ', error);
}
});
}
}