pythonpandasftpftplib

Reading files from FTP server to DataFrame in Python

I would like to load a file from an FTP server into Pandas dataframe without downloading it to disk first. I have written a script that executes this command but with downloading to disk. Is this possible in the ftplib library? Do you see any solution to this problem?

from ftplib import FTP
import os
import pandas as pd
ftps = FTP('gssc.esa.int')
ftps.login()
ftps.cwd('/gnss/data/daily/2019/001/')
filename = '19001.V3status'
local_filename = os.path.join(r"C:/path/where/download/files", filename) #example
lf = open(local_filename, "wb")
ftps.retrbinary('RETR ' + filename, lf.write)
file = "C:/path/where/download/files/" +filename #example
dataV3status = pd.read_fwf(file,
                           names = ('Mon_ID', 'Full_Mon_ID', 'RNX_Ver.', 'Dly(H)',
                                    'Dly(M)', 'V', 'Receiver_Type', 'Antenna_Type',
                                    'Mkr_Name', 'Marker_Number', 'Typ', 'G', 'R',
                                    'E', 'C', 'J', 'S', 'I', 'MD5_Checksum'), 
                           widths = [5,9,5,5,6,2,20,22,5,10,3,3,2,2,2,2,2,2,32],
                           header = None,
                           skiprows = 5,
                           skipfooter = 16)
Solution

  • If you want to stick with ftplib, you can do something like this:

    from io import BytesIO

flo = BytesIO()
ftp.retrbinary('RETR ' + filename, flo.write)
flo.seek(0)
pd.read_fwf(flo, ...)

    Though, pandas.read_fwf documentation claims that it supports FTP directly.
    So this should do too:

    pd.read_fwf("ftp://gssc.esa.int//gnss/data/daily/2019/001/19001.V3status", ...)