I want to run and debug my own C++ extensions for python in "hybrid mode" in VSCode. Since defining your own python wrappers can be quite tedious, I want to use pybind11 to link C++ and python. I love the debugging tools of vscode, so I would like to debug both my python scripts as well as the C++ functions in vscode.
Fortunately, debugging python and C++ files simultaneously is possible by first starting the python debugger and then attach a gdb debugger to that process as described in detail in nadiah's blog post (Windows users, please note this question). This works fine for me. Unfortunately, they define the C++ -- python bindings manually. I would like to use pybind11 instead.
I created a simplified example that is aligned with nadiah's example using pybind11. Debugging the python file works but the gdb debugger doesn't stop in the .cpp file. According to this github question it should be theoretically possible but there are no details on how to achieve this.
Here I try to follow nadiahs example as closely as possible but include pybind11 wrappers.
Create a virtual environment (also works with anaconda, as described below)
virtualenv --python=python3.8 myadd
cd myadd/
. bin/activate
Create file myadd.cpp
#include <pybind11/pybind11.h>
float method_myadd(float arg1, float arg2) {
float return_val = arg1 + arg2;
return return_val;
}
PYBIND11_MODULE(myadd, handle) {
handle.doc() = "This is documentation";
handle.def("myadd", &method_myadd);
}
, myscript.py
import myadd
print("going to ADD SOME NUMBERS")
x = myadd.myadd(5,6)
print(x)
and setup.py
from glob import glob
from distutils.core import setup, Extension
from pybind11.setup_helpers import Pybind11Extension
def main():
setup(name="myadd",
version="1.0.0",
description="Python interface for the myadd C library function",
author="Nadiah",
author_email="nadiah@nadiah.org",
ext_modules=[Pybind11Extension("myadd",["myadd.cpp"])],
)
if __name__ == "__main__":
main()
Clone the pybind11 repo
git clone git@github.com:pybind/pybind11.git
and install the python package
pip install pybind11
Run the setup script
python3 setup.py install
Now, we can already run the python script
python myscript.py
Open vscode
code .
Select the python interpreter with Ctrl+Shift+p -> Select python interpreter -> ./bin/python, now in the lower bar, you should see virtualenv myadd.
Create the launch.json file by clicking the debug symbol and 'Create new launch configuration'.
This is my launch.json (This might be the problem)
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python",
"type": "python",
"request": "launch",
"program": "myscript.py",
"console": "integratedTerminal"
},
{
"name": "(gdb) Attach",
"type": "cppdbg",
"request": "attach",
"program": "${workspaceFolder}/bin/python", /* My virtual env */
"processId": "${command:pickProcess}",
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
],
"additionalSOLibSearchPath": "${workspaceFolder}/build/lib.linux-x86_64-3.8;${workspaceFolder}/lib;${workspaceFolder}/lib/python3.8/site-packages/myadd-1.0.0-py3.8-linux-x86_64.egg/"
}
]
}
Note that I added the "additionalSOLibSearchPath" option in accordance to the github question but it did not change anything.
In vscode, add breakpoints in myscript.py in line 5 and 7, and in myadd.cpp in line 5.
Now, first start the python debugger and let it stop on the breakpoint in line 5.
Then go to a terminal and get the correct process id of the running python script.
ps aux | grep python
The second to last process is the correct one in my case. E.g.
username **65715** 3.0 0.0 485496 29812 pts/3 Sl+ 10:37 0:00 /home/username/myadd/bin/python /home/username/.vscode/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy --connect 127.0.0.1:38473 --configure-qt none --adapter-access-token ... myscript.py
In this example, 65715 would be the correct process id.
Now, in vscode start the (gdb) Attach debugger and type in the process id in the search bar. Hit enter, now you need to type y in the console to allow the attaching and type in your sudo password.
If you are following nadiah's example, you can now press continue on the python debug bar and the script will stop on the C++ breakpoint.
For this pybind11 example, the script does not stop on the C++ breakpoint.
Your project structure now should look like this
myadd
| bin/
| build/
| dist/
| lib/
| myadd.cpp
| myadd.egg-info/
| myscript.py
| pybind11/
| setup.py
As stated in the github post, one has to ensure that the debug flag is set. Therefore, I added a setup.cfg file
[build_ext]
debug=1
[aliases]
debug_install = build_ext --debug install
And ran
python setup.py debug_install
but this did not help as well.
Using conda instead of virtualenv is quite easy. Just create your env as usual and then type in
which python
to get the path to the python executable. Replace the "program" in the (gdb) Attach debug configuration of your launch.json with this path.
I run
I think the C++ code was not build with debug information.
Adding the keyword argument extra_compile_args=["-g"] to the Pybind11Extension in the setup.py may be enough to solve it.
Regardless read on for my solution proposal, that worked for me.
I could make this work by using the Python C++ Debugger extension by BeniBenj, by setting the C++ flag -g and by using the --no-clean pip flag. For the sake of completeness, I am going to enclose here my minimal working project.
Create the bindings in the add.cpp file:
#include <pybind11/pybind11.h>
float cpp_add(float arg1, float arg2) {
float return_val = arg1 + arg2;
return return_val;
}
PYBIND11_MODULE(my_add, handle) {
handle.doc() = "This is documentation";
handle.def("cpp_add", &cpp_add);
}
Create the testing python script:
import my_add
if __name__ == "__main__":
x = 5
y = 6
print(f"Adding {x} and {y} together.")
z = my_add.cpp_add(x, y)
print(f"Result is {z}")
Create the setup.py file:
import os
from distutils.core import setup
from pybind11.setup_helpers import Pybind11Extension
setup(name="myadd",
version="1.0.0",
ext_modules=[Pybind11Extension("my_add", ["add.cpp"], extra_compile_args=["-g"])],
)
The important thing about the setup.py file is that it builds the C++ code with debug information. I have the suspicion that this is what was missing for you.
The package can be installed with:
pip install --no-clean ./
The --no-clean is important. It prevents the sources that your debugger will try to open from being deleted.
Now is the time for launching both the Python and the C++ debuggers. I am using the Python C++ Debugger extension by BeniBenj as recommended by the creator in a Github issue. After installing it, just create a debug config by clicking on the "create a launch.json file", selecting "Python C++ Debugger" and than choosing from the options. (For me both the Default and the Custom: GDB worked.)
Place the breakpoints in both the python and the C++ code. (In the python code, I recommend placing them on the line with the binded code and the one after it.) Select your script and run the "Python C++ Debugger" configuration. The code should pause on entry, and on the second terminal that just opened this question should appear:
Superuser access is required to attach to a process. Attaching as superuser can potentially harm your computer. Do you want to continue? [y/N]
Answer y. Start debugging. Upon reaching your binded code in python, you may have to click manually in the call stack (in the debug panel on the left) to actually switch into the C++ code.
I could not find online how to set the compiler flags within the setup.py.
Therefore I looked into the source code of the Pybind11Extension. There I saw this line:
env_cppflags = os.environ.get("CPPFLAGS", "")
This suggested that I can set the flags with environment variables, and indeed I could. I added this line in the setup.py before the setup() function:
os.environ["CPPFLAGS"] = "-g"
However, as I was typing this answer I also saw this comment in the pybind11 source code:
# flags are prepended, so that they can be further overridden, e.g. by
# ``extra_compile_args=["-g"]``.
I tested it and it works too. However, I could not find it in the documentation. That is the main reason I am including these steps here.
For me. when the debugger pauses on the breakpoint within the C++ code, it wants to open the source file within tmp/pip-req-build-o1w6len6/add.cpp. If the --no-clean option is not kept in the pip installation, then this file will not be found. (I had to create it and copy the source code into it.)
Here is the Custom: GDB configuration, where the "miDebuggerPath" parameter may be deleted:
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python C++ Debugger",
"type": "pythoncpp",
"request": "launch",
"pythonLaunchName": "Python: Current File",
"cppAttachName": "(gdb) Attach"
},
{
"name": "(gdb) Attach",
"type": "cppdbg",
"request": "attach",
"program": "/home/dudly01/anaconda3/envs/trash_env/bin/python",
"processId": "",
"MIMode": "gdb",
"miDebuggerPath": "/path/to/gdb or remove this attribute for the path to be found automatically",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
]
},
{
"name": "Python: Current File",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal"
}
]
}
Here is the default configuration:
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python C++ Debugger",
"type": "pythoncpp",
"request": "launch",
"pythonConfig": "default",
"cppConfig": "default (gdb) Attach"
}
]
}