I am trying to reduce the stack size for a python script in order to produce more threads. I am setting it like this
threading.stack_size(32768)
And putting it at the import level, before any threads get started The problem is that, it has no effect. With or without it, i can start the exact number of threads. I am using Python 2.7 x86 on a Windows 7 machine x64, over 2.4GB RAM available.
Any thoughts on why it has no effect ? I am really looking for a solution, rather than comments on the number of threads I need to spawn.
I suspect it has to do with the .pyc files related to threading, I remember i faced this situation where deleting thoose .pyc files and letting python recompile them on the next run would apply the changes, but I do not know what files to temper with regarding threading.
Would appreciate any help I can get.
P.S. Please do not recommend using Twisted or Asyncore, I know they are better, but I need to make the best of it regarding the current situation, rather than to redesign the entire code.
The minimum thread stack size on Windows is probably at least 64kB. Quoting:
The operating system rounds up the specified size to the nearest multiple of the
system's allocation granularity (typically 64 KB). To retrieve the allocation
granularity of the current system, use the GetSystemInfo function.
So trying to set it to 32kB will probably look a lot like trying to set it to 64kB.
Additionally, CPython implements threading.stack_size on Windows so that it only controls the initially committed stack. It does not try to control the reserve memory for the stack. From the same location:
The reserved memory size represents the total stack allocation in virtual memory.
This means that each of your threads uses up the reserve memory size in virtual memory. You didn't mention how many threads you manage to create before encountering an error but I suspect it's just enough to exhaust addressable memory in your process (which is probably a 32 bit process even though you're running it on Windows 7 x86-64 because the CPython build/distribution is x86(-32)).
That is, even though you (your threads) are not using the memory and even though you have more physical memory on the system, Python can't address the extra memory with its tiny 32 bit pointers and so new threads beyond the limit you're encountering can't have their reserve memory allocated (because there are no addresses left to assign to it).
If you want to be able to change the per-thread reserved memory then you probably need to call CreateThread or _beginthreadex differently than CPython calls it. This probably means need to get CPython changed.
That said, and at the risk of having you yell at me, I seriously doubt you need more than the 1500 threads you can already create.