I'm building a Linux module, on a Debian distribution, using DKMS, and I have a kernel freeze that I need to debug.
I've installed the crash command line tool, with the kernel debug symbols.
When a freeze happens, I have a crash file which I load with the crash command, and I can see the filenames and line numbers for each of the lines from the kernel trace, but when it is about my module, I don't have any filename, nor line number.
Here is an example:
WARNING: active task ffff98d215239e80 on cpu 1 not found in PID hash
KERNEL: /usr/lib/debug/boot/vmlinux-4.19.0-8-amd64
DUMPFILE: /var/crash/202004292108/dump.202004292108 [PARTIAL DUMP]
CPUS: 12
DATE: Wed Apr 29 21:07:00 2020
UPTIME: 00:13:32
LOAD AVERAGE: 0.62, 0.57, 0.37
TASKS: 886
NODENAME: debian
RELEASE: 4.19.0-8-amd64
VERSION: #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)
MACHINE: x86_64 (3700 Mhz)
MEMORY: 16 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000018"
PID: 15775
COMMAND: "douaned"
TASK: ffff98d215239e80 [THREAD_INFO: ffff98d215239e80]
CPU: 1
STATE: EXIT_DEAD (PANIC)
crash> bt -l
PID: 15775 TASK: ffff98d215239e80 CPU: 1 COMMAND: "douaned"
#0 [ffff98d36e843870] machine_kexec at ffffffff87656967
./debian/build/build_amd64_none_amd64/./arch/x86/include/asm/mem_encrypt.h: 75
#1 [ffff98d36e8438c8] __crash_kexec at ffffffff877213ad
./debian/build/build_amd64_none_amd64/./kernel/kexec_core.c: 958
#2 [ffff98d36e843990] crash_kexec at ffffffff877221fd
./debian/build/build_amd64_none_amd64/./include/linux/compiler.h: 219
#3 [ffff98d36e8439a8] oops_end at ffffffff8762a0ad
./debian/build/build_amd64_none_amd64/./arch/x86/kernel/dumpstack.c: 334
#4 [ffff98d36e8439c8] no_context at ffffffff876653ee
./debian/build/build_amd64_none_amd64/./arch/x86/mm/fault.c: 808
#5 [ffff98d36e843a20] __do_page_fault at ffffffff87665ab2
./debian/build/build_amd64_none_amd64/./arch/x86/mm/fault.c: 1323
#6 [ffff98d36e843a90] page_fault at ffffffff87e0114e
/build/linux-AqT5hv/linux-4.19.98/arch/x86/entry/entry_64.S: 1204
[exception RIP: idr_find]
RIP: ffffffff87d0b860 RSP: ffff98d36e843b40 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff98d20e848000 RCX: 0000000000000000
RDX: 000000009456dad9 RSI: 0000000000005807 RDI: 0000000000000008
RBP: ffff98d363bc3ae8 R8: 000000000001421d R9: 00000000917bc5e0
R10: ffff98d36e843bae R11: 0000000000000049 R12: 0000000000000004
R13: ffff98d226491480 R14: ffff98d206f28cfc R15: ffff98d206f28d10
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
./debian/build/build_amd64_none_amd64/./lib/idr.c: 177
#7 [ffff98d36e843b40] find_get_pid at ffffffff8769e846
./debian/build/build_amd64_none_amd64/./include/linux/pid.h: 71
#8 [ffff98d36e843b48] netfiler_packet_hook at ffffffffc1962eda [douane]
#9 [ffff98d36e843ce8] nf_hook_slow at ffffffff87c2eea4
./debian/build/build_amd64_none_amd64/./net/netfilter/core.c: 512
#10 [ffff98d36e843d18] __ip_local_out at ffffffff87c3c8a7
./debian/build/build_amd64_none_amd64/./include/linux/netfilter.h: 248
#11 [ffff98d36e843d70] ip_local_out at ffffffff87c3c927
./debian/build/build_amd64_none_amd64/./net/ipv4/ip_output.c: 123
...
Have a look at the line #8.
From the DKMS documentation, I found the STRIPE=[#] config to which I passed no in order to not remove the debug symbols (as the documentation says), but I still don't have any filenames nor line numbers.
Can anyone tell how can I have them showing please?
I finally figured it out !
You first need to know where is installed your kernel module file (douane.ko in my case, which is installed in /lib/modules/$(uname -r)/updates/dkms/), then after having loaded the crash file, like it is shown in the question, you then need to load the module's debug symbols using the mod command:
crash> mod -s douane /lib/modules/4.19.0-8-amd64/updates/dkms/douane.ko
MODULE NAME SIZE OBJECT FILE
ffffffffc17f8040 douane 28672 /lib/modules/4.19.0-8-amd64/updates/dkms/douane.ko
From now on, using bt -l will show you also the line number from your kernel module source code too.