clojurecore.asynccider

Clojure program works fine when debugged, fails in repl


I'm learning core.async and have written a simple producer consumer code:

(ns webcrawler.parallel
  (:require [clojure.core.async :as async
             :refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout]]))

(defn consumer
  [in out f]
  (go (loop [request (<! in)]
        (if (nil? request)
          (close! out)
          (do (print f)
            (let [result (f request)]
              (>! out result))
              (recur (<! in)))))))

(defn make-consumer [in f]
  (let [out (chan)]
    (consumer in out f)
    out))

(defn process
  [f s no-of-consumers]
  (let [in (chan (count s))
        consumers (repeatedly no-of-consumers #(make-consumer in f))
        out (async/merge consumers)]
    (map #(>!! in %1) s)
    (close! in)
    (loop [result (<!! out)
           results '()]
      (if (nil? result)
        results
        (recur (<!! out)
               (conj results result))))))

This code works fine when I step in through the process function in debugger supplied with Emacs' cider.

(process (partial + 1) '(1 2 3 4) 1)
(5 4 3 2)

However, if I run it by itself (or hit continue in the debugger) I get an empty result.

(process (partial + 1) '(1 2 3 4) 1)
()

My guess is that in the second case for some reason producer doesn't wait for consumers before exiting, but I'm not sure why. Thanks for help!


Solution

  • The problem is that your call to map is lazy, and will not run until something asks for the results. Nothing does this in your code.

    There are 2 solutions:

    (1) Use the eager function mapv:

    (mapv #(>!! in %1) items)
    

    (2) Use the doseq, which is intended for side-effecting operations (like putting values on a channel):

    (doseq [item items]
      (>!! in item))
    

    Both will work and produce output:

    (process (partial + 1) [1 2 3 4] 1) => (5 4 3 2)
    

    P.S. You have a debug statement in (defn consumer ...)

    (print f)
    

    that produces a lot of noise in the output:

    <#clojure.core$partial$fn__5561 #object[clojure.core$partial$fn__5561 0x31cced7
    "clojure.core$partial$fn__5561@31cced7"]>
    

    That is repeated 5 times back to back. You probably want to avoid that, as printing function "refs" is pretty useless to a human reader.

    Also, debug printouts in general should normally use println so you can see where each one begins and ends.