Packet drops on node networking - QEMU/KVM

Hannan

Hi All,

We have some issues with the networking thought you may have the same experience or you could help.
We are running QEMU/KVM on Dual Epyc servers (256 logical core) with Mellanox dual 40G connections. When our customers were pushing more than 50k pps and a few hundred mbps starts to drop the VM's packets. However, the server load and the network load are very low almost nothing.

We were start tune the network and the Kernel of the host as AMD/Intel recommended to see how it goes. Started using Vhost and multi-queues but now we are seeing a heavy load on the CPU from the Vhost/network which is higher than 70% of the CPU time with only a small number of VM's on these servers. Around 60VM's with low usage on CPU and 500Mbit (110k pps) on the network. The two Epyc 7h12 with 256 logical cores are pushing around 80 percent of the CPU and more than 60 percent of it is from the network/vhost. Now we don't have packet drops but under a heavy CPU load.

Here are the configurations and changes we made.

We have started with this 3 months ago

-netdev tap,id=nic0,ifname=$TAPIF,script=no,downscript=no,vhost=on,queues=8 \ -device virtio-net-pci,netdev=nic0,mq=on,vectors=10,mac=$MAC
Still had packet loss

Latest config. High CPU load but no packet drops.
-netdev tap,id=nic0,ifname=$TAPIF,script=no,downscript=no,vhost=on,queues=2,poll-us=50000 \ -device virtio-net-pci,netdev=nic0,mq=on,vectors=10,rx_queue_size=1024,tx_queue_size=1024,packed=on,mac=$MAC

This is how we create a newtork interface
ip tuntap add $TAPIF mode tap user $َUSERNAME multi_queue

Any idea?

Thanks

emgh

I've seen a thread just like this but I just can't remember who/where/when (it was on here though).

Calin

@emgh https://lowendtalk.com/discussion/comment/3509095

emgh

@Calin said:
@emgh https://lowendtalk.com/discussion/comment/3509095

That's it!

Hannan

Seems the solution is DPDK. We are working on it and will update you the results here.