What I need, is to copy a .csv file from an SFTP server to a Google Cloud Storage bucket without downloading the file locally.
Here is my code so far:
from google.cloud import storage
import io
import paramiko
# connection parameters
host = 'host'
port = 22
usr = 'me'
pwd = '0123'
# Connect to the SFTP server
sftp_client = paramiko.SSHClient()
sftp_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sftp_client.connect(host, port, usr, pwd)
sftp = sftp_client.open_sftp()
# Get the file
sftp_file = sftp.open('./file.csv', 'rb')
# Call the Google Cloud Storage API
storage_client = storage.Client()
bucket_name = 'the_bucket'
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(sftp_file)
# Upload to bucket
blob.upload_from_filename(sftp_file)
And here is the error returned:
ValueError: <paramiko.sftp_file.SFTPFile object at 0x0000022665279A00> could not be converted to unicode
Can someone unstuck me?
Use Blob.upload_from_file
to upload the file-like object that you have opened using SFTPClient.open
:
with sftp.open('./file.csv', 'rb') as sftp_file:
sftp_file.prefetch()
blob.upload_from_file(sftp_file)
For the purpose of SFTPFile.prefetch
, see Reading file opened with Python Paramiko SFTPClient.open method is slow.
And just to be very clear, this code still downloads the file from SFTP server to the local machine – it just does not store it to a local file.