python: httplib.CannotSendRequest when nesting threaded SimpleXMLRPCServers

I am intermittently receiving a httplib.CannotSendRequest exception when using a chain of SimpleXMLRPCServers that use the SocketServer.ThreadingMixin.

What I mean by 'chain' is the following:

I have a client script which uses xmlrpclib to call a function on a SimpleXMLRPCServer. That server, in turn, calls another SimpleXMLRPCServer. I realise how convoluted that sounds, but there are good reasons that this architecture has been selected, and I don't see a reason it shouldn't be possible.

(testclient)client_script ---calls--> 
    (middleserver)SimpleXMLRPCServer ---calls---> 
        (finalserver)SimpleXMLRPCServer --- does something

• If I do not use SocketServer.ThreadingMixin then this issue doesn't occur (but I need the requests to be multi-threaded so this doesn't help.)
• If I only have a single level of services (ie just client script calling final server directly) this doesn't happen.

I have been able to reproduce the issue in the simple test code below. There are three snippets:

finalserver:

import SocketServer
import time
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler

class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass

# Create server
server = AsyncXMLRPCServer(('', 9999), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()

def waste_time():
    time.sleep(10)
    return True

server.register_function(waste_time, 'waste_time')
server.serve_forever()

middleserver:

import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib

class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass

# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()

s = xmlrpclib.ServerProxy('http://localhost:9999')
def call_waste():
    s.waste_time()
    return True

server.register_function(call_waste, 'call_waste')
server.serve_forever()

testclient:

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:8888')
print s.call_waste()

To reproduce, the following steps should be used:

1. Run python finalserver.py
2. Run python middleserver.py
3. Run python testclient.py
4. While (3) is still running, run another instance of python testclient.py

Quite often (almost every time) you will get the error below the first time you try to run step 4. Interestingly, if you immediately try to run step (4) again the error will not occur.

Traceback (most recent call last):
  File "testclient.py", line 6, in <module>
    print s.call_waste()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<class 'httplib.CannotSendRequest'>:">

The internet appears to say that this exception can be caused by multiple calls to httplib.HTTPConnection.request without intervening getresponse calls. However, the internet doesn't discuss this in the context of SimpleXMLRPCServer. Any pointers in the direction of resolving the httplib.CannotSendRequest issue would be appreciated.

=========================================================================================== ANSWER:

Okay, I'm a bit stupid. I think I was staring at the code for too protracted a period of time that I missed the obvious solution staring me in the face (quite literally, because the answer is actually in the actual question.)

Basically, the CannotSendRequest occurs when an httplib.HTTPConnection is interrupted by an intervening 'request' operation. Each httplib.HTTPConnection.request must be paired with a .getresponse() call. If that pairing is interrupted by another request operation, the second request will produce the CannotSendRequest error. so:

connection = httplib.HTTPConnection(...)
connection.request(...)
connection.request(...)

will fail because you have two requests on the same connection before any getresponse is called.

Linking that back to my question:

1. the only place in the three programs where such connections are being made are in the serverproxy calls.
2. the problem only occurs during threading, so it's likely a race condition.
3. the only place a serverproxy call is shared is in middleserver.py

The solution then, is obviously to have each thread create it's own serverproxy. The fixed version of middleserver is below, and it works:

import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib

class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass

# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()

def call_waste():
    # Each call to this function creates its own serverproxy.
    # If this function is called by concurrent threads, each thread
    # will safely have its own serverproxy.
    s = xmlrpclib.ServerProxy('http://localhost:9999')
    s.waste_time()
    return True

server.register_function(call_waste, 'call_waste')
server.serve_forever()

Since this version results in each thread having its own xmlrpclib.serverproxy, there is no risk of the same instance of serverproxy invoking HTTPConnection.request more than once in succession. The programs work as intended.

Sorry for the bother.

Okay, I'm a bit stupid. I think I was staring at the code for to protracted a period of time that I missed the obvious solution staring me in the face (quite literally, because the answer is actually in the actual question.)

Basically, the CannotSendRequest occurs when an httplib.HTTPConnection is interrupted by an intervening 'request' operation. Basically, each httplib.HTTPConnection.request must be paired with a .getresponse() call. If that pairing is interrupted by another request operation, the second request will produce the CannotSendRequest error. so:

connection = httplib.HTTPConnection(...)
connection.request(...)
connection.request(...)

will fail because you have two requests on the same connection before any getresponse is called.

Linking that back to my question:

1. the only place in the three programs where such connections are being made are in the serverproxy calls.
2. the problem only occurs during threading, so it's likely a race condition.
3. the only place a serverproxy call is shared is in middleserver.py

The solution then, is obviously to have each thread create it's own serverproxy. The fixed version of middleserver is below, and it works:

import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib

class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass

# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()

def call_waste():
    # Each call to this function creates its own serverproxy.
    # If this function is called by concurrent threads, each thread
    # will safely have its own serverproxy.
    s = xmlrpclib.ServerProxy('http://localhost:9999')
    s.waste_time()
    return True

server.register_function(call_waste, 'call_waste')
server.serve_forever()

Since this version results in each thread having its own xmlrpclib.serverproxy, there is no risk of the serverproxy invoking HTTPConnection.request more than once in succession. The programs work as intended.

Sorry for the bother.