clojuretransducer

Can I transduce / educe over multiple collections without concatting them


Below is a bare-bones version of what I'm doing:

(eduction (map inc) (concat [1 2] [3 4]))
; -> (2 3 4 5)

Is there a way to get the same eduction, without having to pay the cost of concat, which creates an intermediate lazy seq?

The following would perhaps already be a bit less wasty, as instead of the lazy seq, we just have a vector, but I wonder if even that can be avoided.

(eduction (comp cat (map inc)) [[1 2] [3 4]])

Solution

  • It may be simplest to process your collections separately and combine the results. There is, in fact, an easy reducers-based solution that does exactly that under the covers.

    The clojure.core.reducers namespace has cat, a combining function for fold, that you can repurpose to construct a reducible concatenation of your vectors.

    (require '[clojure.core.reducers :as r])
    
    (eduction (map inc) (r/cat [1 2] [3 4]))
    ;; => (2 3 4 5)
    

    This avoids the lazy sequence used in concat. If you have more than two vectors, you can concatenate them all with (reduce r/cat [] colls) or similar.

    This approach did speed up some of the experiments I did, though not your particular example.