rust

Execute concurrent async functions on each element of the hashmap using rayon::par_iter with for_each


Can I do something like this ?

I want to execute concurrent async functions on each element of the hashmap.

async fn b() {
  let hm: HashMap<String, String> = ...;
   
  hm.par_iter().for_each(async move |(k, v)| {
    // ...operations on k,v

    another_func(&v).await;
  })
}
Error on async move:

`async move {` rustc E0658
mismatched types

expected unit type `()`
 found opaque type `impl std::future::Future` rustc E0308

mod.rs(61, 43): the found opaque type

Solution

  • rayon is not a good fit for async because it operates on ordinary non-async functions, and blocks in operations like for_each(). But since you're using async, you could avoid Rayon altogether and just spawn a task for each operation:

    let tasks: Vec<_> = hm
        .iter()
        .map(|(k, v)| {
            let k = k.clone();
            let v = v.clone();
            tokio::spawn(async move {
                some_func(&k, &v);
                another_func(&v).await;
            })
        })
        .collect();
    future::join_all(tasks).await;
    

    Playground

    Since tasks are lightweight, and are executed in parallel on a multi-threaded executor, this is in principle equivalent to using Rayon.

    Note that we clone the keys and the values because that allows them to be safely moved into the task. Without the clone the borrow checker cannot prove that a reference to the data in the hash table passed to a global task won't outlive the hash table itself.