Proxmox Hypervisor and Guest Network Perfomance

Die_Quelle

Hello Guys

I am experiencing a really slow network.

Hypervisor (Hetzner Dedicated with 200mbits guaranteed connection)

root@host ~ # ./speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Hetzner Online GmbH (xxxxxxxx)...
Selecting best server based on latency...
Hosted by HillCom Solutions (Alsfeld) [33.58 km]: 13.244 ms
Testing download speed........................................
Download: 591.59 Mbit/s
Testing upload speed..................................................
Upload: 193.99 Mbit/s

KVM VM (with virtio network)

root@host ~ # ./speedtest-cli
    Retrieving speedtest.net configuration...
    Retrieving speedtest.net server list...
    Testing from Hetzner Online GmbH (xxxxxxxx)...
    Selecting best server based on latency...
    Hosted by HillCom Solutions (Alsfeld) [33.58 km]: 8.29 ms
    Testing download speed........................................
    Download: 94.86 Mbit/s
    Testing upload speed..................................................
    Upload: 77.66 Mbit/s

VM OS is an ipfire firewall (Up2Date)

Any ideas?

casualjoe

I've not used virtio but could it be a 100mbit virtual nic?

AshleyUk

What does virsh dumpxml kvmid show under the network card section

Die_Quelle

@casualjoe said:
I've not used virtio but could it be a 100mbit virtual nic?

How can I check this?

lspci:
00:12.0 Ethernet controller: Red Hat, Inc Virtio network device

@AshleyUk said:
What does virsh dumpxml kvmid show under the network card section

I cant find such command? How can I check this in the KVM VM?

casualjoe

virsh is a host node command not guest. Try it there.

Die_Quelle

@casualjoe said:
virsh is a host node command not guest. Try it there.

there is no virsh only pvesh and then its like an Windows cmd no dumpxml and kvmid command...

AshleyUk

@Die_Quelle said:
there is no virsh only pvesh and then its like an Windows cmd no dumpxml and kvmid command...

Not used their custom OS install before, however a google shows the XML files for the guests should be in : /etc/pve/qemu-server

If you can give us the output of the file/network section.

Verelox

Not sure what might be the issue; but just to check, could you post the output of ethtool eth0 on your KVM?

(i.e ethtool INTERFACE replacing INTERFACE with your network interface)

Die_Quelle

@AshleyUk said:

bootdisk: ide0
cores: 8
cpu: host
ide0: local:100/vm-100-disk-1.qcow2,format=qcow2,size=25G
ide2: local:iso/ipfire-2.17.i586-full-core89.iso,media=cdrom
memory: 2048
name: firewall
net0: virtio=00:50:56:00:80:75,bridge=vmbr0
net2: virtio=3A:FB:97:3D:2C:30,bridge=vmbr2
onboot: 1
ostype: l26
sockets: 1
startup: order=1

My Networking is configured:

I got 2 IPv4 Adresses (external)

1 for Host
1 for Firewall (Ipfire)

Ipfire got 2 interfaces red0 and green0 (normal configuration)

The Networks are bridged:

ethtool -K of all 4 interfaces (1 unused)

http://pastebin.com/snn1yjJ1

I disabled gro and gxo on all interfaces

MarkTurner

If you place a file on the host server eg 10GB file accessible via apache, if you wget that from the VM, what speed do you get.

If that is equally low then its likely your virtual network interface is rate limited.

Verelox

@Die_Quelle said:
ethtool -K of all 4 interfaces (1 unused)

How about ethtool eth0? The reason I'm asking is because the supported link modes might be able to give some hints of why the speed is low compared to the host machine; ethtool (not ethtool -k, this only shows the features) contains this information as well as the duplex and speed.

Die_Quelle

@Verelox said:
How about ethtool eth0? The reason I'm asking is because the supported link modes might be able to give some hints of why the speed is low compared to the host machine; ethtool (not ethtool -k, this only shows the features) contains this information as well as the duplex and speed.

Ahhh, this is what I searched too:

Settings for **eth0**:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes

Verelox

Looks good, but I don't think that's VirtIO; though I might be wrong. Could you please run this last command? It should tell exactly what driver this is:

ethtool -i eth0

Here's an output from a VM running VirtIO:

driver: virtio_net
version: 1.0.0
firmware-version:
bus-info: 0000:00:03.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Die_Quelle

@Verelox this is the hostmaschine.

On a vm it only outputs this:

Settings for eth0:
    Link detected: yes

with -i:

driver: virtio_net
version: 1.0.0
firmware-version: 
bus-info: 0000:00:12.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Verelox

Thank you, I don't think it would be the network card in this case; maybe it's something else. Did you try @MarkTurner 's suggestion? I have seen that ipfire requires special network configuration to operate, so I would personally retry the same speed test on a VM running CentOS to ensure that the host machine is configured properly: if speed improvements is noted then I would think that the issue might be with the network configuration on the VM running ipfire; I'm really out of other ideas however but I do hope that you get the issue solved! Sorry if I couldn't be of much help here.

MarkTurner

Test VM->Host next then you can either eliminate it or confirm thats the cause.

Die_Quelle

I just configured a new vm with debian 8 directly to the internet (ipfire place) and i got this:

so this must be a ipfire related issue. But why is it slowing so down so hard.

MarkTurner

http://wiki.ipfire.org/en/configuration/network/proxy/wui_conf/limit_b

Die_Quelle

@MarkTurner said:
http://wiki.ipfire.org/en/configuration/network/proxy/wui_conf/limit_b

sorry but, this is just proxy related, this service is off.
regards

Thanks for all your help, i hope i can fix this

MarkTurner

Are there any tc rules ratelimiting the interface?

Die_Quelle

@MarkTurner said:
Are there any tc rules ratelimiting the interface?

Yes there are:

qdisc fq_codel 8009: dev red0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 8002: dev green0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 800a: dev imq0 root refcnt 2 limit 10240p flows 1024 quantum 16000 target 5.0ms interval 100.0ms ecn 

What does they mean?

Die_Quelle

I deleted those and finally, this vm is going places. /edit or not, this is really unstable. Its really random

Hosted by Base-Mail (Frankfurt) [100.73 km]: 18.702 ms
Testing download speed........................................
Download: 714.72 Mbit/s
Testing upload speed..................................................
Upload: 234.82 Mbit/s

This must be a bug...

MarkTurner

Die_Quelle said: What does they mean?

It means there is some rate limiting going on.

I think its more likely that the TC classes are not setup yet. Just have a look at the rate limiting link I sent you and setup the rate limiting as you need it.