pythonmultithreadingftplib

Using ftplib for multithread uploads


I'm trying to do multithread uploads, but get errors. I guessed that maybe it's impossible to use multithreads with ftplib?

Here comes my code:

    class myThread (threading.Thread):
    def __init__(self, threadID, src, counter, image_name):
        self.threadID = threadID
        self.src = src
        self.counter = counter
        self.image_name = image_name
        threading.Thread.__init__(self)
    def run(self):
        uploadFile(self.src, self.image_name)

def uploadFile(src, image_name):
    f = open(src, "rb")            
    ftp.storbinary('STOR ' + image_name, f)
    f.close()

ftp = FTP('host')   # connect to host, default port
ftp.login()               # user anonymous, passwd anonymous@   
dirname = "/home/folder/"
i = 1   
threads = []

for image in os.listdir(dirname):
    if os.path.isfile(dirname + image):
        thread = myThread(i , dirname + image, i, image )   
        thread.start()
        threads.append( thread )        
        i += 1  

for t in threads:
    t.join()

Get bunch of ftplib errors like

raise error_reply, resp error_reply: 200 Type set to I

If I try to upload one by one, everything works fine


Solution

  • Have you tried to put the connection code inside the thread?

    In other words, make each thread do their own separate connection with FTP.host() and FTP.login(). The server may not like multiple uploads at the same time on a single connection, because it may be parsing commands one at a time and can't handle a second upload or "STOR" command. But if you can do multiple connections from the same IP address, you'll have separate session on which to issue the 'STOR' command.

    Here's an example:

        class myThread (threading.Thread):
            def __init__(self, threadID, src, counter, image_name):
                 ###############
                 #Add ftp connection here!
                 self.ftp = FTP('host')   # connect to host, default port
                 self.ftp.login()               # user anonymous, passwd anonymous@   
                 ################
                 self.threadID = threadID
                 self.src = src
                 self.counter = counter
                 self.image_name = image_name
                 threading.Thread.__init__(self)
            def run(self):
                 uploadFile(self.src, self.image_name)
    
        def uploadFile(src, image_name):
              f = open(src, "rb")            
              self.ftp.storbinary('STOR ' + image_name, f)
              f.close()
    
         dirname = "/home/folder/"
         i = 1   
         threads = []
    
         for image in os.listdir(dirname):
              if os.path.isfile(dirname + image):
                 thread = myThread(i , dirname + image, i, image )   
                 thread.start()
                 threads.append( thread )        
                 i += 1  
    
          for t in threads:
              t.join()
    

    See if that behaves better.

    P.S. Not sure if all my tabs are aligned.