amazon-s3sftpsshdminaapache-mina

Apache Mina SSHd SFTP server - get file size being uploaded in advance to forward it to S3 using multipart upload


As far as I understand from the standard which Apache MINA SSHd SFTP server is based on here, it means that it is not necessary to send information about the size of the file transferred beforehand (servers should not require it) - in other words, we never know the size of the file being transferred until we're done with the transfer OR the client used sends it as a file metadata. Is this correct? How to read this metadata? Which SFTP clients send that piece of information?

Those questions are important to tackle the problem of transferring larger files to S3 bucket used as an underlying storage of SFTP. If you don't know the size in advance you cannot do the multipart upload so you have to first store the file in its entirety and only then perform the actual upload - which causes the SFTP session timeout...


Solution

  • Which SFTP clients send that piece of information?

    WinSCP does.

    PSFTP/FileZilla and OpenSSH sftp do not.

    If you don't know the size in advance you cannot do the multipart upload

    I do not think you are correct about that. There's nothing in the S3 CreateMultipartUpload request about the actual file size or number of upload parts.