I am trying to determine why my KVM host shows high system CPU usage for a specific guest.
I have setup a KVM host (Ubuntu 20.04) to host a guest VM (Ubuntu 20.04). I configured the guest to use cores (via vcpu / vcpupin / emulatorpin) from the processor in socket 1, memory (via numatune) connected to channels on the processor in socket 1, and the NIC connected (via interface / address source) to the channels on the processor in socket 1 to ensure the VMs run in the most optimal state. Even though I have gone through the process of configure IOMMU, SR-IOV, and Intel VF to pass Virtual Function PCI devices for the NICs directly into the VMs, the host instance still sees high system cpu usage on the cores where the guest has parked it's NIC interrupts.
I cannot figure out why the host system is involved, since the PCI device is disconnected from the host and attached directly to the guest. This causes huge performance issues. Has any experienced this? Is there a configuration error on my end? Please see the configurations and mpstat data below:
<vcpu placement='static' cpuset='0,2,4,6,8,10,12,14,16,18,20,22,24,52,54,56,58,60,62,64,66,68,70,72,74,76'>26</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='4'/>
<vcpupin vcpu='3' cpuset='6'/>
<vcpupin vcpu='4' cpuset='8'/>
<vcpupin vcpu='5' cpuset='10'/>
<vcpupin vcpu='6' cpuset='12'/>
<vcpupin vcpu='7' cpuset='14'/>
<vcpupin vcpu='8' cpuset='16'/>
<vcpupin vcpu='9' cpuset='18'/>
<vcpupin vcpu='10' cpuset='20'/>
<vcpupin vcpu='11' cpuset='22'/>
<vcpupin vcpu='12' cpuset='24'/>
<vcpupin vcpu='13' cpuset='52'/>
<vcpupin vcpu='14' cpuset='54'/>
<vcpupin vcpu='15' cpuset='56'/>
<vcpupin vcpu='16' cpuset='58'/>
<vcpupin vcpu='17' cpuset='60'/>
<vcpupin vcpu='18' cpuset='62'/>
<vcpupin vcpu='19' cpuset='64'/>
<vcpupin vcpu='20' cpuset='66'/>
<vcpupin vcpu='21' cpuset='68'/>
<vcpupin vcpu='22' cpuset='70'/>
<vcpupin vcpu='23' cpuset='72'/>
<vcpupin vcpu='24' cpuset='74'/>
<vcpupin vcpu='25' cpuset='76'/>
<emulatorpin cpuset='0,2,4,6,8,10,12,14,16,18,20,22,24,52,54,56,58,60,62,64,66,68,70,72,74,76'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
<cpu mode='host-passthrough' check='none'/>
<interface type='hostdev' managed='yes'>
<mac address='52:54:08:06:48:01'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x19' slot='0x02' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:08:06:48:01'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x19' slot='0x0a' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:08:06:48:02'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x19' slot='0x02' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:08:06:48:02'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x19' slot='0x0a' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
</interface>
22:26:38 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
22:26:43 all 0.00 0.00 3.39 0.00 0.00 0.09 0.00 7.30 0.00 89.22
22:26:43 0 0.00 0.00 7.30 0.00 0.00 0.00 0.00 26.57 0.00 66.13
22:26:43 1 0.00 0.00 0.00 0.00 0.00 8.43 0.00 0.00 0.00 91.57
22:26:43 2 0.00 0.00 10.38 0.00 0.00 0.00 0.00 29.34 0.00 60.28
22:26:43 4 0.00 0.00 8.69 0.00 0.00 0.00 0.00 29.09 0.00 62.22
22:26:43 6 0.00 0.00 5.00 0.00 0.00 0.00 0.00 23.20 0.00 71.80
22:26:43 8 0.20 0.00 39.31 0.00 0.00 0.00 0.00 35.03 0.00 25.46
22:26:43 10 0.00 0.00 6.96 0.00 0.00 0.00 0.00 31.21 0.00 61.83
22:26:43 12 0.00 0.00 44.90 0.00 0.00 0.00 0.00 35.31 0.00 19.80
22:26:43 14 0.00 0.00 13.18 0.00 0.00 0.00 0.00 32.05 0.00 54.77
22:26:43 16 0.20 0.00 5.31 0.00 0.00 0.00 0.00 26.12 0.00 68.37
22:26:43 18 0.00 0.00 15.95 0.00 0.00 0.20 0.00 30.47 0.00 53.37
22:26:43 20 0.00 0.00 7.04 0.00 0.00 0.00 0.00 26.16 0.00 66.80
22:26:43 22 0.00 0.00 15.76 0.00 0.00 0.00 0.00 31.92 0.00 52.32
22:26:43 24 0.00 0.00 7.35 0.00 0.00 0.00 0.00 27.96 0.00 64.69
22:26:43 52 0.00 0.00 10.46 0.00 0.00 0.00 0.00 29.78 0.00 59.76
22:26:43 54 0.00 0.00 7.35 0.00 0.00 0.00 0.00 26.12 0.00 66.53
22:26:43 56 0.00 0.00 6.91 0.00 0.00 0.00 0.00 29.67 0.00 63.41
22:26:43 58 0.00 0.00 7.82 0.00 0.00 0.00 0.00 29.42 0.00 62.76
22:26:43 60 0.00 0.00 37.17 0.00 0.00 0.00 0.00 35.15 0.00 27.68
22:26:43 62 0.00 0.00 14.99 0.00 0.00 0.00 0.00 28.54 0.00 56.47
22:26:43 64 0.00 0.00 37.50 0.00 0.00 0.00 0.00 33.53 0.00 28.97
22:26:43 66 0.00 0.00 7.30 0.00 0.00 0.00 0.00 29.61 0.00 63.08
22:26:43 68 0.00 0.00 8.08 0.00 0.00 0.00 0.00 27.88 0.00 64.04
22:26:43 70 0.00 0.00 7.09 0.00 0.00 0.00 0.00 24.49 0.00 68.42
22:26:43 72 0.20 0.00 6.94 0.00 0.00 0.00 0.00 29.96 0.00 62.90
22:26:43 74 0.00 0.00 7.63 0.00 0.00 0.00 0.00 27.91 0.00 64.46
22:26:43 76 0.00 0.00 8.63 0.00 0.00 0.00 0.00 29.32 0.00 62.05
22:26:43 98 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 99.80
22:26:43 100 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 99.80
Does anyone know what is configured incorrectly or why the %sys is so high on cores 8, 12, 60, 64?
This ended up being an issue with the default kvm.halt_poll_ns value. Setting /sys/module/kvm/parameters/halt_poll_ns to 0 caused this problem to stop happening.