linuxcompressiongziptarbzip2

Picking compression algorithm for low-memory device - what works without being entirely in memory?


I'm writing an app that, on a low-memory device, needs to be able to read the list of contained files and also extract specific (but random) files from a tarball (or some sort of archive).

Here's my question: can a bz2 tarball be "streamed" as it's decompressed (where "stream" means that I can read the list of files or extract a random file without reading the entire tarball into memory)?

What about for a gz tarball?

Are there any formats that are better or worse for this?

I have really tried to find this info online but I'm having trouble.


Solution

  • Yes. But you need to define "low" in "low-memory". All commonly-used compression formats stream. Though they also all have some amount of memory required for history information, independent of the length of the entire stream. E.g. ~4MB for bzip2, ~32K for zlib/gzip.