I am trying to write a Python program using a client-server model that supports uploading a file from client to server by fragmenting the file and for each fragment, create a thread to send it.
I try to fragment the file into 32 KB chunks.
def __init__(self, host, port, threads=5):
self.host = host
self.port = port
self.fragment_size = 1024 * 32
self.threads = threads
def upload_file(self):
file_name = filedialog.askopenfilename(title="Select file to upload")
if file_name:
with ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.submit(self.send_file, file_name)
def send_file(self, filename):
try:
file_size = os.path.getsize(filename)
fragment_count = (file_size // self.fragment_size) + (1 if file_size % self.fragment_size != 0 else 0)
if not self.send_metadata(filename, file_size, fragment_count):
raise Exception("Failed to send metadata.")
with open(filename, 'rb') as file:
with ThreadPoolExecutor(max_workers=self.threads) as executor:
futures = []
for fragment_number in range(fragment_count):
fragment = file.read(self.fragment_size)
futures.append(executor.submit(self.send_fragment, fragment, fragment_number, fragment_count))
for future in futures:
future.result()
except Exception as e:
messagebox.showerror("Error", f"Failed to upload file: {e}")
And for each send_fragment thread, I create a new socket:
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client_socket.connect((self.host, self.port))
The code seems to work correctly with small files, but when it comes to large files (larger than 500MB), this error occurs:
[WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted.
The requirement of this program is to fragment the file and create a thread to send each fragment.
Every time you call connect() on a socket, by default it will bind to a random local ephemeral port that is available. When the socket is closed, it may take the OS a few moments to release the port. If you run netstat on the command-line, you will likely see a lot of ports in TIME_WAIT state. That happens while the OS is holding the ports open to flush out any pending packets on the network.
Having send_fragment() use a new socket means that sending a 500+ MB file fragmented into 32K chunks will use 16000+ TCP connections. Creating and closing so many connections in a short amount of time will easily exhaust the range of available ephemeral ports.
To address this, you could try throttling the threads to introduce some delay after each connection is closed, so the local ports have more time to be released.
But a better design is to instead reuse your TCP connections. Send multiple fragments per connection (you will have to delimit your fragments so the server can identify where one fragment ends and the next begins). Put the sockets into a pool, and have each thread poll a socket from that pool when needed. When a fragment is done sending, don't close its socket, put it back in the pool. Don't create and connect a new socket unless the pool is empty.
Since you are running only 5 threads at a time, you should be able to send all 16000+ fragments using just 5 TCP connections rather than 16000+ connections.
If you don't want to write your own socket pool, there are 3rd party socket pools available for Python, for instance this pool and this pool.