[SOLVED] kretprobe handlers in kernel 3.x not getting called

kretprobe handlers in kernel 3.x not getting called

I have been learning and experimenting with kernel probes(both kprobes and kretprobes) recently and today I managed to make it work for Ubuntu 22.04 kernel 5.x(which is the kernel I use on my development machine) and Debian 12.5 kernel 6.x which is a VM I have and it worked without any issue.

I started to check some older kernels 3.x and 4.x, so I installed CentOS 7 with kernel 3.10.0.1160.119.el7.x86_64...

This is where the problem start. First of all the naming of the syscalls was different in kernel 3.x. For example __x64_sys_getdents64 is now sys_getdents64(there is also SyS_getdents64 I really don't know the difference).

[user@localhost ~]$ cat /proc/kallsyms | grep getdents
0000000000000000 T SyS_getdents
0000000000000000 T sys_getdents
0000000000000000 T SyS_getdents64
0000000000000000 T sys_getdents64
0000000000000000 T compat_sys_getdents
0000000000000000 T compat_sys_getdents64
...

So I changed the code and installed the kretprobe on sys_getdents64. I could install kernel probes successfully according to the return address of register_kretprobe but my handlers are not getting called. To narrow down the problem, I wrote a simplified version of my code,

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>

static int entry_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
    printk(KERN_INFO "entry_handler: getdents64 called\n");
    return 0;
}

static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
    printk(KERN_INFO "ret_handler: getdents64 returned\n");
    return 0;
}

static struct kretprobe my_kretprobe = {
    .handler = ret_handler,
    .entry_handler = entry_handler,
    .maxactive = 20,
};

static int __init kretprobe_init(void)
{
    int ret;

    my_kretprobe.kp.symbol_name = "sys_getdents64";
    ret = register_kretprobe(&my_kretprobe);
    if (ret < 0) {
        printk(KERN_INFO "register_kretprobe failed, returned %d\n", ret);
        return -1;
    }
    printk(KERN_INFO "Planted return probe at %s: %p\n",
           my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);
    return 0;
}

static void __exit kretprobe_exit(void)
{
    unregister_kretprobe(&my_kretprobe);
    printk(KERN_INFO "kretprobe at %p unregistered\n", my_kretprobe.kp.addr);
}

module_init(kretprobe_init)
module_exit(kretprobe_exit)
MODULE_LICENSE("GPL");

and building it with following Makefile,

obj-m += lkmtest.o

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

I get the following output from the Makefile,

 make
make -C /lib/modules/3.10.0-1160.119.1.el7.x86_64/build M=/home/user/test modules
make[1]: Entering directory `/usr/src/kernels/3.10.0-1160.119.1.el7.x86_64'
  CC [M]  /home/user/test/lkmtest.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/user/test/lkmtest.mod.o
  LD [M]  /home/user/test/lkmtest.ko
make[1]: Leaving directory `/usr/src/kernels/3.10.0-1160.119.1.el7.x86_64'

after insmod lkmtest.ko I do get the following confirmation in dmesg

[  225.686827] Planted return probe at SyS_getdents64: ffffffffa3c722e0

But no matter what I do I don't see the message in printk of kretprobe handler! I tried ls of different directories, run top but no sign of any message from probe handler in dmesg logs.

I have checked and it seems kprobes are enabled on this kernel,

[root@localhost user]# cat /sys/kernel/debug/kprobes/enabled
1

Weird enough, I tested the example code from https://github.com/spotify/linux/blob/master/samples/kprobes/kretprobe_example.c and the kretprobe on do_fork worked nicely, but when I changed the func_name variable to getdents64 it had the same behavior as I stated above.

I don't have any goal to make any product or something. I'm just using my free time at work to learn about linux kernel and linux kernel internals and experiment.

After trying different stuff to limit of my knowledge and some articles on the web I couldn't spot the issue myself so I'm here to get some professional help.

Solution

Since I was on 64bit Linux I was under the assumption that I should use the syscalls ending with 64, for example SyS_getdents64 or sys_getdents64.

ffffffffb28721c0 T SyS_getdents
ffffffffb28721c0 T sys_getdents
ffffffffb28722e0 T SyS_getdents64
ffffffffb28722e0 T sys_getdents64

But I was wrong... by using sys_getdents I could get my handler functions to get called! I still do not understand the logic but this is how my problem was solved.