rustrust-tokioreqwest

Copy reqwest bytes_stream in to tokio file


I'm attempting to copy a file downloaded with reqwest in to a tokio file. This file is too large to store in memory so it needs to be through the bytes_stream() rather than bytes()

I had attempted the following

let mut tmp_file = tokio::fs::File::from(tempfile::tempfile()?);
let byte_stream = reqwest::get(&link).await?.bytes_stream();
tokio::io::copy(&mut byte_stream, &mut tmp_file).await?;

This fails due to

    |
153 |     tokio::io::copy(&mut byte_stream, &mut tmp_file).await?;
    |     --------------- ^^^^^^^^^^^^^^^^ the trait `tokio::io::AsyncRead` is not implemented for `impl Stream<Item = Result<bytes::bytes::Bytes, reqwest::Error>>`
    |     |
    |     required by a bound introduced by this call

Is there any way I can get the trait AsyncRead on the Stream or otherwise copy this data in to the file? The reason I'm using a tokio file is I later need to AsyncRead from it. Perhaps it would make sense to copy in to a regular std::File and then convert it to a tokio::fs::File?


Solution

  • This method works. Loosely based on the example for bytes_stream()

    Note: You probably want to buffer the writes here

    use futures::StreamExt;
    
    let mut tmp_file = tokio::fs::File::from(tempfile::tempfile()?);
    let mut byte_stream = reqwest::get(&link).await?.bytes_stream();
    
    while let Some(item) = byte_stream.next().await {
      tokio::io::copy(&mut item?.as_ref(), &mut tmp_file).await?;
    }