I have a C++/Qt application where I am running a Python interpreter session within the process of the main application. I have built a 'python console' as a QPlainTextEdit widget, which handles the input and output for the interpreter using the Python C API. The point of all this is that Python will have direct access to data that I'm holding inside the main application. All is well, except that I want to be able to end the current interpreter session and restart it without exiting the main application.
Currently, I'm trying the obvious way:
Py_Initialize();
// Run the main session...
Py_FinalizeEx();
// Restart the session
Py_Initialize();
From reading other posts and Python's documentation, there are possibly issues with reloading certain modules after finalizing the interpreter due to memory leaks. This seems to be true in my case: re-importing certain modules (like 'numpy') but not others (like 'sys') will trigger an exception and fail.
Is there a good workaround strategy to restart the interpreter without these issues? For instance, if I spawn a sub-interpreter, will finalizing it and restarting a new sub-interpreter run into the same problems I'm having? The alternative I was trying to avoid is to run Python out-of-process, which I think would allow me to restart by basically killing the process and starting a new one. Any strategy advice would be much appreciated.
From Py_FinalizeEx documentation:
Bugs and caveats: The destruction of modules and objects in modules is done in random order; this may cause destructors (del() methods) to fail when they depend on other objects (even functions) or modules. Dynamically loaded extension modules loaded by Python are not unloaded. Small amounts of memory allocated by the Python interpreter may not be freed (if you find a leak, please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.
Specifically note that:
Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.
(emphasys is mine)
It is clear that the design does not support finalization and reinitialization as you attempt.
It is therefore recommended that you initialize the python engine once (when your process starts or at the first time you need it), and only finalize it upon exit (or at least when you are sure you will not need it anymore).
Even if you find some workaround that will work in a specific scenario I do not think it is a good idea given the information above.
Note:
If you have some requirement that mandates creating a new Python envirionment for each session or operation, the best approach is to run Python as an external process.