javazipsftpbatch-processing

Stream compressed chunks of data to SFTP server

I'm loading batches of data from database and need to save them to a remote SFTP server using Java into a single bzip2 compressed file. I'm using JSch SFTP client. Is there any possibility to compress each batch (chunk) of data on the fly and concatenate to a resulting file? Or is it possible to use BufferedInputStream and to compress the buffer on every iteration and pass it to put(InputStream src, String dst) method of JSch? I don't want to store all the data in memory or local file because of its size. Is it possible with bzip2 format?

Solution

  • Use one of the ChannelSftp.put overloads that return OutputStream:
    https://epaul.github.io/jsch-documentation/javadoc/com/jcraft/jsch/ChannelSftp.html#put-java.lang.String-

    And write the bzip data to the stream as you create it.

    Or pipe bzip OutputStream to an InputStream that you pass to JSch:
    How do I convert an OutputStream to an InputStream?