clinuxdebugginglinux-kernelpanic

How to read, understand, analyze, and debug a Linux kernel panic?


Consider the following Linux kernel dump stack trace; e.g., you can trigger a panic from the kernel source code by calling panic("debugging a Linux kernel panic");:

[<001360ac>] (unwind_backtrace+0x0/0xf8) from [<00147b7c>] (warn_slowpath_common+0x50/0x60)
[<00147b7c>] (warn_slowpath_common+0x50/0x60) from [<00147c40>] (warn_slowpath_null+0x1c/0x24)
[<00147c40>] (warn_slowpath_null+0x1c/0x24) from [<0014de44>] (local_bh_enable_ip+0xa0/0xac)
[<0014de44>] (local_bh_enable_ip+0xa0/0xac) from [<0019594c>] (bdi_register+0xec/0x150)

Solution

  • It's just an ordinary backtrace. Those functions are called in reverse order; the first function listed was called by the next. Below I added [was called] to further clarify:

    (unwind_backtrace+0x0/0xf8) [was called] from (warn_slowpath_common+0x50/0x60)
    (warn_slowpath_common+0x50/0x60) [was called] from (warn_slowpath_null+0x1c/0x24)
    (warn_slowpath_null+0x1c/0x24) [was called] from (local_bh_enable_ip+0xa0/0xac)
    (local_bh_enable_ip+0xa0/0xac) [was called] from (bdi_register+0xec/0x150)
    

    The bdi_register+0xec/0x150 is the symbol + the offset/length. There's more information about this in Understanding a Kernel Oops and how you can debug a kernel oops. Also, there's this excellent tutorial on Debugging the Kernel.

    Note: as suggested below by Eugene, you may want to try addr2line first. It will need an image with debugging symbols though. For example:

    addr2line -e vmlinux_with_debug_info 0019594c(+offset)