RX mbuf alloc failed queue_id=0 when trying to use DPDK with UHD

I'm trying to use DPDK with UHD. Things work fine without DPDK.

I think the problem is with DPDK 24.11 with UHD 4.7 (master branch) on Ubuntu 24.04LTS.

I tried other versions combinations.

# dpdk-devbind --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=

Hugepages seems to be setup:

# dpdk-hugepages.py --show
Node Pages Size Total
0 2048 2Mb 4Gb

Huge page filesystems mounted at: /dev/hugepages
# cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 2048
HugePages_Free: 2046
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 4194304 kB

(I've tried 1GB hugepages too)

uhd.conf is setup per the example in the link.

[use_dpdk=1]
dpdk_mtu=9000
dpdk_driver=/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-25.0/
dpdk_corelist=2,3,4
dpdk_num_mbufs=4095
dpdk_mbuf_cache_size=315

[dpdk_mac=xxxxx]
dpdk_lcore = 3
dpdk_ipv4 = 192.168.30.1/24
dpdk_num_desc = 4096

[dpdk_mac=xxxx]
dpdk_lcore = 4
dpdk_ipv4 = 192.168.40.1/24
dpdk_num_desc = 4096

I'm running the bencharmark_rate example which works without DPDK.

# /usr/local/lib/uhd/examples/benchmark_rate --rx_rate 250e6 --rx_subdev "A:0 B:0" --rx_channels 0,1 --tx_rate 250e6 --tx_subdev "A:0 B:0" --tx_channels 0,1 --args "addr=192.168.30.2,second_addr=192.168.40.2,mgmt_addr=x.x.x.x,master_clock_rate=250e6,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 13.3.0; Boost_108300; DPDK_24.11; UHD_4.7.0.0-210-ge75629c3
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
IXGBE_INIT: ixgbe_alloc_rx_queue_mbufs(): RX mbuf alloc failed queue_id=0
IXGBE_INIT: ixgbe_dev_rx_queue_start(): Could not alloc mbuf for queue:0
IXGBE_INIT: ixgbe_dev_start(): Unable to start rxtx queues
IXGBE_INIT: ixgbe_dev_start(): failure in ixgbe_dev_start(): -1
[ERROR] [DPDK] Port 1: Could not start device
EAL: already called initialization.
[ERROR] [UHD] Device discovery error: RuntimeError: DPDK: Failure to start device
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization
[00:00:00.000617] Creating the usrp device with: addr=192.168.30.2,second_addr=192.168.40.2,mgmt_addr=129.218.156.110,master_clock_rate=250e6,use_dpdk=1...
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL initialization
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.30.2
second_addr: 192.168.40.2
mgmt_addr: x.x.x.x
master_clock_rate: 250e6
use_dpdk: 1

EAL: Error disabling MSI-X interrupts for fd 32

The final message doesnt always appear depending which version of UHD/DPDK I try

I think the main error is:

IXGBE_INIT: ixgbe_alloc_rx_queue_mbufs(): RX mbuf alloc failed queue_id=0
IXGBE_INIT: ixgbe_dev_rx_queue_start(): Could not alloc mbuf for queue:0
IXGBE_INIT: ixgbe_dev_start(): Unable to start rxtx queues
IXGBE_INIT: ixgbe_dev_start(): failure in ixgbe_dev_start(): -1

I tried on another computer and also in a docker container and get the same error.

The only other thing I can think of is the pre-requisites mentioned in https://doc.dpdk.org/guides/nics/ixgbe.html I mentions some driver features but I dont know how to check/change them.

Solution

First and foremost, dpdk_num_mbufs=4095 is wittingly smaller than dpdk_num_desc=4096, which might be the reason behind the Rx queue start error displayed. Also, this somehow does not account for Tx mbufs (should such at all be needed in the OP's use case).

Consider to use value of dpdk_num_mbufs that is greater than dpdk_num_desc and prefer a power of 2, say, 8192. Alternatively, one may decrease the number of Rx queue descriptors (dpdk_num_desc=2048). Also, consider to set dpdk_mbuf_cache_size either to, say, 512 or to 0 (no cache).