I'm trying to use Sub-interpreter for having distinct environment, and having multi-thread on same environment(interpreter).
However, when I tried to cleanup sub-interpreter, Python raise Fatal error with message:
Py_EndInterpreter: thread still has a frame
Here is the minimal reproducing code:
#include <Python.h>
#include <thread>
using namespace std;
void child(PyThreadState* ts) {
PyEval_RestoreThread(ts);
PyRun_SimpleString("print('hello world')");
PyEval_SaveThread();
}
int main() {
Py_Initialize();
PyThreadState* main_ts = PyThreadState_Get();
PyThreadState* sub_ts = Py_NewInterpreter();
PyEval_SaveThread();
thread t1(child, sub_ts);
thread t2(child, sub_ts);
t1.join();
t2.join();
PyEval_RestoreThread(sub_ts);
Py_EndInterpreter(sub_ts);
PyThreadState_Swap(main_ts);
Py_Finalize();
}
Currently, I figure out two workarounds:
1. create a new thread under sub-interpreter
After spawning sub-interpreter, create a new thread state and use it
new_ts = PyThreadState_New(sub_ts->interp);
Before calling Py_EndInterpreter(), clear and delete the thread by
PyThreadState_Clear(new_ts); PyThreadState_Delete(new_ts);
2. Delete sub-interpreter manually
Since Py_EndInterpreter()
will check lots of things, we can remove interpreter manually by:
Note: I afraid this method may cause memory leak.
However, I am curious what is wrong with my original answer? Is it related to the Py_NewInterpreter() that the doc has mention that
Note that no actual thread is created
You are passing a single thread state to two threads and trying to do "restore" in both, simultaneously. Intuitively, that seems like a Bad Idea. Each Python thread has a lot of state to track, like the stack and the local variables, which need to be different for each thread.
This has a lot of good information about this rather obscure topic:
https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock