clojurehttp-kit

Clojure http-kit get request stuck on multiple async calls


I have created small example which highlight the problem:

(->> (range 0 4)
     (mapv (fn [i]
             (http/get "http://http-kit.org/"
                       (fn [res]
                         (info "first callback")
                         (let [res2 @(http/get "http://http-kit.org/")]
                           (info "second callback ")))))))

It's stuck on printing the 4s first callback msg's.

enter image description here

If I change the range for 0..3 it will work, the sync version also works.

Update:

The (info) is a taoensso.timbre logging library


Solution

  • It looks like the issue caused because the http-kit client thread pool doesn't relieve the thread until callback function is finished. And so it ends up with run out of threads.

    So I started to think how to make callback function quicker and came up with this solution:

    I have created async wrapper function for http-kit client get which using clojure.core.async/chan in callback to quickly put the result in channel and then wait for it results and execute the heavy callback:

    (defn async-http-get
      [url opts callback]
      (let [channel (chan)]
        (http/get url opts #(go (>! channel %)))
        (go (callback (<! channel)))
        nil))
    

    So now using async-http-get instead of http/get solve the problem for me.

    (->> (range 0 4)
         (mapv (fn [i]
                 (async-http-get "http://http-kit.org/"
                           (fn [res]
                             (info "first callback")
                             (let [res2 @(http/get "http://http-kit.org/")]
                               (info "second callback ")))))))
    

    enter image description here