intelqemukvmlibvirtnic

Get SR-IOV Virtual Function counters


Suppose I've got SR-IOV passthrough enabled on the host with 2 Virtual Functions, I'm running two QEMU/KVM VM's with libvirt, each connected to a VF respectively, is there any way to see the VF counters on the host (such as rx/tx pkts)?
I've tried to use ethtool -S to see stats but I can only see the global counters of the physical function.
I found an SR-IOV counters plugin for OpenStack Ceilometer but it's a Mellanox plugin and uses a proprietary drivers on the Guest VM's.
Any help would be appreciated.


Solution

  • I have tested this solution on Ubuntu, running on a SRIOV enabled compute in openstack installation. Steps to get counters for individual VF of a PF. The physical NIC (25G) is running on INTEL i40 driver for XXV710 device.

    #lspci |grep -i ethernet
     3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
    .....
    # ethtool -i eth2
    driver: i40e
    version: 2.7.29
    firmware-version: 7.20 0x80009f02 1.2829.0
    expansion-rom-version: 
    bus-info: 0000:3b:00.1
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes
    

    Step 1-

    find for sriov dir in /sys/devices:

    # find  /sys/devices/ -type d -name sriov 
    
    /sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov
    

    Step 2 : Go to this dir and do ls -lrt. You will see the VF numbers. These numbers are same as seen in output of "ip link show ' command. Do the corelation.

    # find  /sys/devices/ -type d -name sriov  |xargs ls -lrt
    
    /sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov:
    total 0
    drwxr-xr-x 3 root root 0 Jan  9 21:40 7
    drwxr-xr-x 3 root root 0 Jan  9 21:40 15
    drwxr-xr-x 3 root root 0 Jan  9 21:40 5
    drwxr-xr-x 3 root root 0 Jan  9 21:40 13
    drwxr-xr-x 3 root root 0 Jan  9 21:40 3
    drwxr-xr-x 3 root root 0 Jan  9 21:40 11
    drwxr-xr-x 3 root root 0 Jan  9 21:40 1
    drwxr-xr-x 3 root root 0 Jan  9 21:40 8
    drwxr-xr-x 3 root root 0 Jan  9 21:40 6
    drwxr-xr-x 3 root root 0 Jan  9 21:40 14
    drwxr-xr-x 3 root root 0 Jan  9 21:40 4
    drwxr-xr-x 3 root root 0 Jan  9 21:40 12
    drwxr-xr-x 3 root root 0 Jan  9 21:40 2
    drwxr-xr-x 3 root root 0 Jan  9 21:40 10
    drwxr-xr-x 3 root root 0 Jan  9 21:40 0
    drwxr-xr-x 3 root root 0 Jan  9 21:40 9
    

    output of ip link show

    ## ip link show eth2
    4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 3cfdfea85df9 state UP mode DEFAULT group default qlen 1000
        link/ether 3c:fd:fe:a8:5d:f9 brd ff:ff:ff:ff:ff:ff
        vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 11 MAC fa:16:3e:19:8c:05, vlan 1929, spoof checking off, link-state auto, trust on
        vf 12 MAC fa:16:3e:43:88:4f, vlan 1927, spoof checking off, link-state auto, trust on
        vf 13 MAC fa:16:3e:4f:b3:d0, vlan 1925, spoof checking off, link-state auto, trust on
        vf 14 MAC fa:16:3e:56:b6:4a, vlan 1923, spoof checking off, link-state auto, trust on
        vf 15 MAC 02:f1:f1:04:01:03, vlan 1938, spoof checking off, link-state auto, trust on
    you can do 1 to 1 mapping and identify which is the vf you are interested in. Assume its vf 15.
    

    Step 3- go to dir /.

    # cd 15/stats
    

    Step 4 - do ls and cat all files , except tx_spoofed

    # ls -lrt
    total 0
    -r--r--r-- 1 root root 4096 Jan  9 21:40 tx_errors
    -r--r--r-- 1 root root 4096 Jan  9 21:40 rx_packets
    -r--r--r-- 1 root root 4096 Jan  9 21:40 tx_dropped
    -r--r--r-- 1 root root 4096 Jan  9 21:40 tx_spoofed
    -r--r--r-- 1 root root 4096 Jan  9 21:40 tx_packets
    -r--r--r-- 1 root root 4096 Jan  9 21:40 rx_bytes
    -r--r--r-- 1 root root 4096 Jan  9 21:40 rx_dropped
    -r--r--r-- 1 root root 4096 Jan  9 21:40 tx_bytes
    
    # tail  *
    ==> rx_bytes <==
    108786154191
    
    ==> rx_dropped <==
    0
    
    ==> rx_packets <==
    975217621
    
    ==> tx_bytes <==
    433665695588
    
    ==> tx_dropped <==
    0
    
    ==> tx_errors <==
    0
    
    ==> tx_packets <==
    590997020
    # tail  *
    ==> rx_bytes <==
    108786397727
    
    ==> rx_dropped <==
    0
    
    ==> rx_packets <==
    975220967
    
    ==> tx_bytes <==
    433665771018
    
    ==> tx_dropped <==
    0
    
    ==> tx_errors <==
    0
    
    ==> tx_packets <==
    590998121
    

    These are your counters.

    BR// Sanjeev Tewari