linux-kerneltraceebpfbpftrace

Why are the kernel function names not resolving in the output of kstack()?

I am trying to see the kernel stack with the following bpftrace command:

root@ubuntu:~$ bpftrace -e 'k:vfs_read{@[kstack] = count()}'
Attaching 1 probe...
^C

@[
    0xffffffffa78d2dc1
    0xffffffffa78d306a
    0xffffffffa7604fd7
    0xffffffffa82000a4
]: 5

What should I do to see kernel function names instead of the addresses?

Solution

  • Take a look at /proc/kallsyms. If the first column is all zeros, it probably means that there are some restrictions placed on exposing kernel addresses. Here's a sample output:

    root@ubuntu:~# head /proc/kallsyms
0000000000000000 A fixed_percpu_data
0000000000000000 A __per_cpu_start
0000000000000000 A cpu_debug_store
0000000000000000 A irq_stack_backing_store
0000000000000000 A cpu_tss_rw
0000000000000000 A gdt_page
0000000000000000 A exception_stacks
0000000000000000 A entry_stack_storage
0000000000000000 A espfix_waddr
0000000000000000 A espfix_stack

    The restrictions can be dropped by setting /proc/sys/kernel/kptr_restrict to 0 (consider 1 as well to keep some of the protections on):

    root@ubuntu:~# echo 0 > /proc/sys/kernel/kptr_restrict

    At least in my case, this was enough to get the addresses to resolve to function names in the output of kstack():

    root@ubuntu:~# bpftrace -e 'k:vfs_read{@[kstack] = count()}'
Attaching 1 probe...
^C

@[
    vfs_read+1
    __x64_sys_read+26
    do_syscall_64+87
    entry_SYSCALL_64_after_hwframe+92
]: 2

    Additional documentation: