raku

Use gather/take with race


I can parallelize execution using Hyper and it works:

$ raku -e 'race for (^8).race(batch => 1, degree => 4) {sleep rand; .say}'
0
3
5
2
1
7
4
6

But how can I add gather/take behavior to such loop?

In this approach take won't detect that it is wrapped in gather context:

$ raku -e '
    my @x = gather race for (^8).race(batch => 1, degree => 4) {
        sleep rand;
        take $_;
    };
    @x.say
'

Died at:
    take without gather

In this approach gather will consume HyperSeq without actually "hypering" over it:

$ raku -e '
    my @x = race gather for (^8).race(batch => 1, degree => 4) {
        sleep rand;
        take $_;
    };
    @x.say
'

[0 1 2 3 4 5 6 7]

Solution

  • Update This wanswer is not necessarily useless. Indeed, at the time I am adding this update @PawelPabianbbkr has accepted it. But @timotimo's answer is outstanding, and I hope that it will become the accepted answer instead of this wanswer.


    I can parallelize execution using race and it works

    But only on Tuesdays. I'm guessing you already know this first bit of my answer, but the compiler is allowed to decide whether to parallelize. To slightly but fairly misquote the relevant design doc with my added emphasis:

    hyper / race request (but do not require) parallel evaluation.

    ----

    But how can I add gather/take behavior to such loop?

    Quoting the design doc again:

    You can think of race as a gather with a 'try take start {...}' on parallel computation.

    That is to say, using race means you already automatically get gather/take behavior for free, without actually having to use gather/take.

    ----

    gather race ... dies with error message take without gather

    race gather ... "works" but doesn't parallelize

    I'm not sure what to make of those two scenarios.

    ----

    In his answer to What concurrency mechanisms are provided by Raku ... ?, Jonathan Worthington mentions this about gather/take:

    • The gather/take mechanism for lazily producing values. This is seen in various languages as generator functions; in Raku, however, there is not a single stack frame limit, so they are coroutine powerful. (This isn't of in any context involving asynchronous programming, but it's still meets the definition of concurrency).

    I'm unsure about the parenthetical. I can't parse "This isn't of in any context involving", and the latter part of the sentence seems ambiguous.

    ----

    One thing that jumped out at me is that gather/take defaults to being lazy. Use of race should override that, and make it eager, but I thought to myself that Rakudo could plausibly have a bug that left it lazy, and wondered if lazy behavior was leading to the behavior seen in your race gather ... code. However, I tried inserting an explicit eager where I thought it might be needed and nothing I tried made a difference.