debugginggdblinux-kernelkernel

thread-aware gdb for the Linux kernel

I am using gdb attached to a serial port of a virtual machine to debug linux kernel.

I am wondering, if there is any patches/plugins which can make the gdb understand some of linux kernel's data structure and make it "thread aware"?

By that I mean under gdb I can see how many kernel threads are there, their status, and for each thread, their stack information.

Solution

  • libvmi

    https://github.com/libvmi/libvmi

    This project does "LibVMI: Simplified Virtual Machine Introspection" which sounds really close.

    This project in particular https://github.com/Wenzel/pyvmidbg uses libvmi and features a demo video of debugging a Windows userland application form inside it, without memory conflicts.

    As of May 2019, there are two limitations however as of May 2019, both of which could be overcome with some work: https://github.com/Wenzel/pyvmidbg/issues/24

    The developer of that project also answered further at: https://stackoverflow.com/a/56369454/895245

    Implementing it with those libraries would be in my opinion the best way to achieve this goal today.

    Linaro lkd-python

    First, this Linaro page claims to have a working setup: https://wiki.linaro.org/LandingTeams/ST/GDB that allows you to do usual thread operations such as thread, bt, etc., but it relies on a GDB fork. I will test it out later. In 2016, https://youtu.be/pqn5hIrz3A8 says that the implementation was in C, not as Python scripts unfortunately, which would be better and avoid forking. The sketch for lkd-python can be found at: https://git.linaro.org/people/lee.jones/kieran.bingham/binutils-gdb.git/log/?h=lkd-python

    Linux kernel in-tree GDB scripts + my brain

    I then tried to see what I could do with the kernel in-tree Python scripts at v4.17 + some manual intervention as a prototype, but didn't quite get there yet.

    I have tested using this highly automated QEMU + Buildroot setup.

    First follow the procedure I described at: How to debug the Linux kernel with GDB and QEMU? to get GDB working.

    Then, as described at: How to debug Linux kernel modules with QEMU? run GDB with:

    gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel

    This loads the in-tree GDB Python scripts from scripts/gdb.

    One of those scripts provides:

    lx-ps

    which lists all threads with format:

    0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd

    The first field is the address of the task_struct struct, so we can see the entire struct with:

    p (struct task_struct)*0xffff88000ed08000

    which should in theory allow us to get any information we want about the process.

    Now I wanted to find the PC. For ARM, I've seen: Find program counter of process in kernel and I tried:

    task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))->uregs[ARM_pc]

    but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: How do I print a #defined constant in GDB? which are apparently not set?

    The dream: a GDB Thread Aware Python extension API

    https://sourceware.org/pipermail/gdb/2017-March/046559.html

    Currently the only way to do it with python I think would be to implement a custom version of every command you want to support:

    But what we really want is to have a GDB api where a python script provides only the minimal necessary parameters for all GDB commands to just work transparently (thread info, thread N and so on).

    The API would basically only need to:

    I think, and everything else might could just work based on those. Then the Linux kernel would be able to maintain its own in-tree Python provider that works for the Linux kernel, and similarly for any other operating systems.