pythonasyncore

too many file descriptors in select() python in windows


i am trying to receive about 1000 connections to my server but it cannot receive more than 512. What can i do to increase the amount of open connections? I am running windows 8.1

Not: I am very new to this stuff so, thanks for help

Here is my code;

import asyncore
import socket
import uuid
import time
import threading

class statistics(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
    def run(self):
        while True:
            entry = raw_input("")
            zaman = int(time.time())
            cmd = receivedCmd
            print "calculating.."
            time.sleep(1)
            if entry == 'istatistik':
                print str(receivedCmd-cmd) + " command/second"
                print "total received commands: " + str(receivedCmd)
                entry = ""
class tcpClient:
    def __init__(self):
        self.clientid = uuid.uuid1(int(time.time()))
        self.buffer = ""
        self.buffer_size = 0
        self.conn_time = time.time()
        self.overflow = 0
        #print str(self.clientid) + " assingned"
    def recv_msg(self, msg):
        global receivedCmd
        self.buffer = msg
        self.buffer_size = len(self.buffer)
        receivedCmd = receivedCmd + 1
        if self.buffer_size >= 1024:
            self.overflow = 1

    def __del__(self):
        print str(self.clientid) + " has left."


class TCPHandler(asyncore.dispatcher_with_send):
    global clist
    def handle_read(self):
        data = self.recv(1024)
        if data:
            if clist[self].overflow:
                self.send("overflow")
                self.handle_close()
            else:
                self.send(data)
                clist[self].recv_msg(data)
    def handle_close(self):
        del clist[self]
        self.close()

    def handle_error(self):
        del clist[self]
        self.close()

class TCPServer(asyncore.dispatcher):
    global clist
    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.set_reuse_addr()
        self.bind((host, port))
        self.listen(5)

    def handle_accept(self):
        self.clist = clist
        pair = self.accept()
        if pair is None:
            pass
        else:
            sock, addr = pair
            #print 'Connection : %s' % repr(addr)
            clist[TCPHandler(sock)] = tcpClient()

if __name__ == '__main__':
    clist = {}
    receivedCmd = 0
    server = TCPServer('', 5000)
    server2 = TCPServer('',5001)

    StaticsThread = statistics()
    StaticsThread.start()
    asyncore.loop()

Note: I still cannot receive more than 512 connections with the Twisted Framework, i don't know what to do. There have to be thousands of connected clients. Please help.


Solution

  • The asyncore module relies in the select OS function, which only supports a limited number of file descriptors.

    As an alternative use a multi-threading server (I won't recommend this) or, better, the Twisted framework which is event-driven (highly recommended!).

    Hope this helps!


    Since Twisted's default reactor under Windows is also select-based then you should consider using the IOCP reactor instead.

    from twisted.internet import iocpreactor
    iocpreactor.install()
    
    from twisted.internet import reactor
    

    But also take into account that Twisted prefers Linux systems (where the default reactor is epoll-based) rather than Windows. Maybe switching to Linux is a better choice.