I have a server using apache2 with mod_python that seems to reset global variables after some short period of time. Until this happens, all the features of the server and global variables are handled exactly as I would expect.
To rule out the possibility of some runtime error (although I do have debugging on and don't see any errors), I made this simplified script to demonstrate how a global variable is being reset. It just holds a global counter and increments it every time the server is accessed:
from mod_python import apache
counter = 0
def handler(req):
global counter
counter += 1
req.content_type = 'text/plain'
req.write('counter: '+str(counter))
return apache.OK
I can keep refreshing the page and watch the counter go up as expected. However, at some point the counter jumps back to 1.
I tried simply holding the F5 key down to rapidly refresh the page and see if it would still reset while I was continually accessing it, and I noticed something else. At first it would still drop down to 1 a few times, but eventually it would drop down to other numbers. For example I got up to around 200 refreshes and it just dropped down to around 100, and would occasionally even jump upward back into the 200s.
It seems like there are multiple virtual shells for the server script being started, and the server just randomly switches between them or starts new ones. How can I prevent this?
It seems like there are multiple virtual shells for the server script being started, and the server just randomly switches between them or starts new ones.
Exactly.
Apache has a variety of different things that it does to maximize concurrency, one of which is to spawn a bunch of child processes.
With mod_python, each child process gets its own independent Python interpreter, which means they each have their own copies of any global variables.
How can I prevent this?
Well, you could disable forking in Apache. But this is a bad idea. Unless you really know what you're doing, this will kill your scalability. And it still won't help if, say, you restart the server (which Apache will sometimes do to itself to clear up memory leaks, unless you configure it not to).
Web services are supposed to be stateless—or, rather, if they have any state, they're supposed to package it up and persist it between requests. For per-user state, you can store it in a cookie, or a hidden form field, and get their browser to pass it back. But for system-wide state, that doesn't work; you need to save it to something external.
The usual solution is a database. But for something this simple, you could use anything—even a plain text file and a flock. Which is verbose, but easy to understand without having to learn SQL or some other database interface, and it makes you think about the concurrency issues (and understanding them is critical), so I'll show that:
import contextlib
import fcntl
from mod_python import apache
@contextlib.contextmanager
def flocking(f, flag=fcntl.LOCK_EX):
fcntl.flock(f, flag)
try:
yield f
finally:
fcntl.flock(f, fcntl.LOCK_UN)
def bump_counter():
while True:
try:
with flocking(open('storage.lock', 'r+')) as f:
val = int(f.read())
f.seek(0)
f.write(str(val))
return val
except OSError:
pass
try:
with flocking(open('storage.lock', 'x')) as f:
val = 0
f.write(str(val))
return val
except OSError:
pass
def handler(req):
req.content_type = 'text/plain'
req.write('counter: '+str(bump_counter()))
return apache.OK
Most of the code is about error handling, mainly to deal with the case that this is the first-ever request, so the file doesn't exist yet. In that case, we try to open it in x mode, which will only succeed if the file doesn't exist, just in case two requests come in right as you start up and they both think they're first. This way, one of them will fail, and will go back and try the whole loop again.
In real life, you'll want better error handling, because "file not found" isn't the only reason this could fail, and you don't want the server to just block forever when that happens. But in real life, you're probably going to use a database that takes care of this for you.