I suspect similar questions might have been asked multiple times.
But it's hard for me to get an answer for my question..
I understand GIL makes only single python thread can execute at a time.
(My understanding comes from https://dabeaz.com/python/UnderstandingGIL.pdf)
But if I think about it, having GIL has essentially the same effect of a single core environment.
With Gil, python multiple threads run in a single core.
As a programmer, I still have to deal with all the race conditions that might occur.
Or is the purpose not about me? it's about python interpreter to be safe?
I guess the question can be rephrased as, if python removes GIL, how is the user program affected?
My understanding is that,
if GIL is gone,
it will make python multithreading:
single core thread programming -> multi core thread programming
Or is there something else going on?
(If my understanding is correct, GIL is something we can actually take out.. I mean if it can be done, it can be done without affecting user programs)
I guess I have to emphasize that here I'm only interested in the user program perspective. (How GIL affects python runtime (interpreter) is not something I ask here)
First, understand that not all Python implementations have a GIL, like Jython and IronPython. However, CPython, the big main one that most people use, does have one.
One thing that's important to note is that CPython has tried to remove the GIL, but it's really hard due to all the stuff built assuming the GIL is there. One quick and easy overview of just some of the problems involved in removing the GIL is described in the PyPy FAQ (PyPy also has a GIL):
Yes, PyPy has a GIL. Removing the GIL is very hard. On top of CPython, you have two problems: (1) GC, in this case reference counting; (2) the whole Python language.
For PyPy, the hard issue is (2): by that I mean issues like what occurs if a mutable object is changed from one thread and read from another concurrently. This is a problem for any mutable type: it needs careful review and fixes (fine-grained locks, mostly) through the whole Python interpreter. It is a major effort, although not completely impossible, as Jython/IronPython showed. This includes subtle decisions about whether some effects are ok or not for the user (i.e. the Python programmer).
CPython has additionally the problem (1) of reference counting. With PyPy, this sub-problem is simpler: we need to make our GC multithread-aware. This is easier to do efficiently in PyPy than in CPython. It doesn’t solve the issue (2), though.
Note that there was work to support a Software Transactional Memory (STM) version of PyPy. This should give an alternative PyPy which works without a GIL, while at the same time continuing to give the Python programmer the complete illusion of having one. This work is currently a bit stalled because of its own technical difficulties.
And all of that's coming from a group of developers that are building their whole new own compiler. Imagine all the issues the CPython group, which is much bigger and needs to ensure a level of backward compatibility too, needs to deal with.
So yes, your guess is right: it's not all about you. If someone can figure out how to take out the GIL without messing up a bunch of other stuff, then it'd probably be merged by CPython. It'd make Python code faster and less confusing to people expecting how threads usually work. But no one's been able to do that good enough yet. From what I've read, it's slowly become less of a "problem we should fix" and more of a "problem we have to deal with".