clojurestreamzipring

Generate and stream a zip-file in a Ring web app in Clojure

I have a Ring handler that needs to:

Now I have it sort of working, but only the first zipped entry gets streamed, and after that it stalls/stops. I feel it has something to do with flushing/streaming that is wrong.

Here is my (compojure) handler:

(GET "/zip" {:as request}
            :query-params [order-id   :- s/Any]
            (stream-lessons-zip (read-string order-id) (:db request) (:auth-user request)))

Here is the stream-lessons-zip function:

(defn stream-lessons-zip
  []
  (let [lessons ...];... not shown

  {:status 200
   :headers {"Content-Type" "application/zip, application/octet-stream"
             "Content-Disposition" (str "attachment; filename=\"files.zip\"")
   :body (futil/zip-lessons lessons)}))

And i use a piped-input-stream to do the streaming like so:

(defn zip-lessons
 "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons]
(let [paths (map #(select-keys % [:file_path :file_name]) lessons)]
(ring-io/piped-input-stream
  (fn [output-stream]
    ; build a zip-output-stream from a normal output-stream
    (with-open [zip-output-stream (ZipOutputStream. output-stream)]
      (doseq [{:keys [file_path file_name] :as p} paths]
        (let [f (cio/file file_path)]
          (.putNextEntry zip-output-stream (ZipEntry. file_name)) 
          (cio/copy f zip-output-stream)
          (.closeEntry zip-output-stream))))))))

So I have confirmed that the 'lessons' vector contains like 4 entries, but the zip file only contains 1 entry. Furthermore, Chrome doesn't seem to 'finalize' the download, ie. it thinks it is still downloading.

How can I fix this?

Solution

  • It sounds like producing a stateful stream using blocking IO is not supported by http-kit. Non-stateful streams can be done this way:

    http://www.http-kit.org/server.html#async

    A PR to introduce stateful streams using blocking IO was not accepted:

    https://github.com/http-kit/http-kit/pull/181

    It sounds like the option to explore is to use a ByteArrayOutputStream to fully render the zip file to memory, and then return the buffer that produces. If this endpoint isn't highly trafficked and the zip file it produces is not large (< 1 gb) then this might work.