I'm downloading an XZ file with hyper, and I would like to save it to disk in decompressed form by extracting as much as possible from each incoming Chunk
and writing results to disk immediately, as opposed to first downloading the entire file and then decompressing.
There is the xz2 crate that implements the XZ format. However, its XzDecoder
does not seem to support a Python-like decompressobj
model, where a caller repeatedly feeds partial input and gets partial output.
Instead, XzDecoder
receives input bytes via a Read
parameter, and I'm not sure how to glue these two things together. Is there a way to feed a Response
to XzDecoder
?
The only clue I found so far is this issue, which contains a reference to a private ReadableChunks
type, which I could in theory replicate in my code - but maybe there is an easier way?
XzDecoder
does not seem to support a Python-like decompressobj model, where a caller repeatedly feeds partial input and gets partial output
there's xz2::stream::Stream
which does exactly what you want. Very rough untested code, needs proper error handling, etc, but I hope you'll get the idea:
fn process(body: hyper::body::Body) {
let mut decoder = xz2::stream::Stream::new_stream_decoder(1000, 0).unwrap();
body.for_each(|chunk| {
let mut buf: Vec<u8> = Vec::new();
if let Ok(_) = decoder.process_vec(&chunk, &mut buf, Action::Run) {
// write buf to disk
}
Ok(())
}).wait().unwrap();
}