Can an eBPF/XDP thread be preempted?

I am currently trying to benchmark some XDP programs. In this benchmark, I'm varying the throughput of the sender packet generator, between 1 and 98Gbps, and I'm also varying the number of cores that will receive the packets, between 1 and 20 cores.

However, I'm facing a strange phenomenon with the following program:

SEC("xdp")
int  simply_drop(struct xdp_md *ctx) {

    int cpu = get_and_check_cpu_id();
    __u64 arrival_time = bpf_ktime_get_ns();
    __u64 finish_time = bpf_ktime_get_ns();
    if(!update_info(arrival_time, finish_time, cpu)) {
        bpf_printk("Error while looking up timer map\n");
    }
    return XDP_DROP;
}

It basically gets two times and saves it to a map, for it to be later used to show the average latency when running things between the two bpf_ktime_get_ns() calls.

But the thing is, when I increase the throughput of the packet generator, the latency decreases. For instance, when I send 1Gbps the latency is around 100ns, while when I send 98Gbps, the latency is 30ns. Additionally, when the number of cores that process arriving packets increases, the latency increases also.

But since this program in specific, doesn't do anything in between the two bpf_ktime_get_ns(), there shouldn't be any contingency between threads, no? Could the threads be preempted in between the two calls?

In any case, I made sure that all cores were located in the same NUMA node.

Solution

It depends on your kernel version. Here is a quote from a BPF mailing list thread on the topic.

Since v5.7 BPF program runners use migrate_disable() instead of preempt_disable(). See commit 2a916f2f546c ("bpf: Use migrate_disable/enable in array macros and cgroup/lirc code.") 2.

But at that time migrate_disable() was merely an alias for preempt_disable() on !CONFIG_PREEMPT_RT kernels.

Since v5.11 migrate_disable() does no longer disable preemption on !CONFIG_PREEMPT_RT kernels. See commit 74d862b682f5 ("sched: Make migrate_disable/enable() independent of RT") 3.

So, yes, you are right, but it depends on the kernel version.

So on newer kernels, XDP programs can be preempted. Though migration is still disabled, so your program will not be scheduled on a different CPU.