Am using python and paramiko to copy a 5GB file between server A and Server B and script will be executed from serverX, which will open a ssh session to serverb from serverX and run the command to copy the file from server B using sshpass. Script is working, but it is not copying the complete 5GB file. it's copying only half and some time less than half.
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(serverb, username=user, password=password)
try:
stdin, stdout, stderr = client.exec_command("sshpass -p password scp -v -r root@serverA:/tmp/file_to_copy_name /tmp/",timeout=None)
except Exception as err:
print("copy between server error")
raise
You may want to use the Rsync over SSH instead of scp (secure remote file copy) with sshpass (noninteractive ssh password provider). It supports the fast incremental file transfer (can resume unfinished upload) and using the SSH key is much more secure than passing the raw password via sshpass.
Something like:
rsync -az /root/bigfile.txt 198.211.117.129:/root/
-a for archive mode
-z to compress file data during the transfer
The manual: https://download.samba.org/pub/rsync/rsync.html
Moreover, it can resume the copy started with scp.
Here is the instruction on how to use it over SSH: https://www.digitalocean.com/community/tutorials/how-to-copy-files-with-rsync-over-ssh
Also, as already pointed out by @pynexj, the client.exec_command() will not wait until the command execution will be finished. So you may want to have some alternative way to check if the file was successfully copied and have the same data as the source. One of the options could be checking the MD5 hash: https://stackoverflow.com/search?q=Python+md5+hash
And you may want to check the: What is the fastest hash algorithm to check if two files are equal?