[SOLVED] Decrease Paramiko maximal request size

Decrease Paramiko maximal request size

I have a Python Runbook and I would like to set the MAX_REQUEST_SIZE = 1024 instead of 32768 (default value) in the Paramiko's sftp_file.py.

Have you any idea about how can we do it??

I'am trying to transfer files from SFTP to local. And for some reasons, the transfer of files with more than 30MB is suspended automatically when achieving the 27-28 MB without getting any error message. When I changed locally the max_request_size to 1024, it worked.

Solution

Try slowing down the transfer instead. A very trivial implementation of transfer throttling can be like:

import time

bytes_per_second = 512*1024
start_time = time.time()

def callback(transferred, total):
    while (transferred / (time.time() - start_time)) > bytes_per_second:
        time.sleep(0.1)

sftp.get(filepath, os.path.join(localFilePath, entry.filename), callback)

Alternatively, just control the request size by re-implementing the download:

with sftp.open(filepath, "rb") as fr, \
     open(os.path.join(localFilePath, entry.filename), "wb") as fl:
    while True:
        data = fr.read(1024)
        fl.write(data)
        if len(data) == 0:
            break

The above is a stripped down version of what SFTPClient.get does internally.