I have created small example which highlight the problem:
(->> (range 0 4)
(mapv (fn [i]
(http/get "http://http-kit.org/"
(fn [res]
(info "first callback")
(let [res2 @(http/get "http://http-kit.org/")]
(info "second callback ")))))))
It's stuck on printing the 4s first callback msg's.
If I change the range for 0..3 it will work, the sync version also works.
Update:
The (info)
is a taoensso.timbre
logging library
It looks like the issue caused because the http-kit client thread pool doesn't relieve the thread until callback function is finished. And so it ends up with run out of threads.
So I started to think how to make callback function quicker and came up with this solution:
I have created async wrapper function for http-kit client get which using clojure.core.async/chan
in callback to quickly put the result in channel and then wait for it results and execute the heavy callback:
(defn async-http-get
[url opts callback]
(let [channel (chan)]
(http/get url opts #(go (>! channel %)))
(go (callback (<! channel)))
nil))
So now using async-http-get
instead of http/get
solve the problem for me.
(->> (range 0 4)
(mapv (fn [i]
(async-http-get "http://http-kit.org/"
(fn [res]
(info "first callback")
(let [res2 @(http/get "http://http-kit.org/")]
(info "second callback ")))))))