I'm attempting to run a python script that spawns a subinterpreter and a main interpreter, each responsible for doing some sort of work (Based on the example used in this article). The work I'm having both interpreters do involves use of the NumPy library, and when I go to run my code, I get the following error:
_xxsubinterpreters.RunFailedError: <class 'ImportError'>: Error importing numpy:
you should not try to import numpy from its source directory; please exit
the numpy source tree, and relaunch your python interpreter from there.
I initially suspected that this had to do with my NumPy install, but found this in the Real Python writeup on Python 3.12 Subinterpreters:
With the GIL moving to be a per-interpreter state, this guaranteed thread safety is removed. To prevent either a large number of bugs in existing extension modules or a lack of adoption of Python 3.12, methods were added so that when you want to import a module, Python can first determine whether it’s ready for this new feature.
And:
Note: If a subinterpreter with a per-interpreter GIL attempts to import a module that doesn’t support this feature, then Python will raise an ImportError.
Is it just the case that the Interpreters module in the Python/C API 3.12 Interpreters module does not yet support NumPy? Otherwise, I'm not sure what the issue might be.
Yes, most C extension modules, including Numpy, don't support multiple sub-interpreters currently. This has to do with:-
Why haven't they enabled multiple sub-interpreter support?
This is because commonly used APIs from C extension modules (e.g. PyGILState_* APIs) are not supported with multiple sub-interpreters. Refer caveats section from Python documentation for more details. Due to this, it isn't wise to allow most C extension modules to run through multiple sub-interpreters currently.
If you want to read up more about this, you can check the linked article I wrote covering my experience using sub-interpreters for parallelism.