rfuturefurrr

Limit of workers in future::plan() function higher than available CPUs cores

I have a CPU of 8 cores which I'm trying to use to do parallel future_map_chr().
My question is as follows: the number of cores in my computer is equal to 8, so how is it possible to parallelize function with plan set as :

plan(multisession, workers = 50,gc=T)

Did I misunderstood the difference between workers and cores ? Is it better than workers = detectCores() ?

Thanks a lot

Solution

  • Author of the Futureverse here:

    Is it better than workers = detectCores()?

    I argue that parallelly::availableCores() is better and safer than parallel::detectCores(), e.g. https://www.jottr.org/2022/12/05/avoid-detectcores/.

    Using:

    plan(multisession)

    is the same as:

    plan(multisession, workers = parallelly::availableCores())

    However, there is nothing in the Futureverse and plan() that prevents you from going over the limit by specifying workers manually as an "AsIs" integer, e.g.

    plan(multisession, workers = I(50))

    So, yes, this will spin up 50 parallel workers in the background, even if you only got 8 CPU cores on your machine.

    EDIT 2024-07-16: The I(.) function, which prepends class AsIs, needs to be used. If not, there will be either a warning or an error if going over the number local CPU core. See https://github.com/HenrikBengtsson/parallelly/issues/107 for more info on this.