Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


XEN Full virtualization -- unstable network: overload?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

XEN Full virtualization -- unstable network: overload?

Hi

I have got new VPS XEN Full virtualization server / 100Mb port
What I am seeing for the third day is low bandwidth rates (lower 5MB/s) and 15ms latency to DE-CIX. Also see large StDevs on every hop -- the network is very unstable, (it is Accelerated based, so I guess uplink is not a problem). Do you think the problem is with the host machine?

==============================================================================================================
Sat Aug  8 01:59:01 GMT-3 2015
+ mtr -rwc10 decix.de
Start: Sat Aug  8 01:59:01 2015
HOST: XXXXXXXXXXXXXXX.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                       0.0%    10    5.8   3.4   0.2   6.7   2.7
  2.|-- 82.211.0.1                          0.0%    10   12.1  10.5   0.3  48.8  14.5
  3.|-- fra3.xe-0-0-0.accelerated.de        0.0%    10   11.7   7.9   0.4  16.9   7.0
  4.|-- sgw2-te-0-0-2-3-ixp.fra.de-cix.net  0.0%    10   13.8   8.4   1.3  21.5   6.8
  5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
+ mtr -rwc10 4.2.2.2
Start: Sat Aug  8 01:59:12 2015
HOST: XXXXXXXXXXXXXXX.de              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                             0.0%    10    6.7   5.8   2.7   8.4   2.1
  2.|-- 82.211.0.1                                0.0%    10    8.0  17.2   8.0  58.1  14.6
  3.|-- xe-4-2-1-311.edge4.Frankfurt1.Level3.net  0.0%    10   13.4  12.4   8.3  16.1   2.5
  4.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- b.resolvers.Level3.net                    0.0%    10   13.8  14.0   9.9  21.3   3.3
+ grep SUM
+ iperf -c ping.online.net -i 1 -P 10 -f M -t 5
[SUM]  0.0- 1.0 sec  7.50 MBytes  7.50 MBytes/sec
[SUM]  1.0- 2.0 sec  5.00 MBytes  5.00 MBytes/sec
[SUM]  2.0- 3.0 sec  4.50 MBytes  4.50 MBytes/sec
[SUM]  3.0- 4.0 sec  4.62 MBytes  4.62 MBytes/sec
[SUM]  4.0- 5.0 sec  4.62 MBytes  4.62 MBytes/sec
[SUM]  0.0- 5.6 sec  27.5 MBytes  4.88 MBytes/sec
+ dd if=/dev/zero of=/root/test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 10.5804 s, 101 MB/s


==============================================================================================================
Sat Aug  8 02:59:01 GMT-3 2015
+ mtr -rwc10 decix.de
Start: Sat Aug  8 02:59:01 2015
HOST: XXXXXXXXXXXXXXX.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                       0.0%    10    0.2   4.6   0.2   7.6   2.3
  2.|-- 82.211.0.1                          0.0%    10    0.8  11.2   0.8  21.8   5.1
  3.|-- fra3.xe-0-0-0.accelerated.de        0.0%    10    0.4  12.6   0.4  17.4   5.1
  4.|-- sgw2-te-0-0-2-3-ixp.fra.de-cix.net  0.0%    10    1.3  13.1   1.3  16.8   4.4
  5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
+ mtr -rwc10 4.2.2.2
Start: Sat Aug  8 02:59:11 2015
HOST: XXXXXXXXXXXXXXX.de              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                             0.0%    10    7.1   3.9   0.1   9.0   3.9
  2.|-- 82.211.0.1                                0.0%    10   16.3   8.7   0.3  16.3   5.7
  3.|-- xe-4-2-1-311.edge4.Frankfurt1.Level3.net  0.0%    10   10.6   7.6   0.6  15.5   5.6
  4.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- b.resolvers.Level3.net                    0.0%    10   12.7   8.2   0.7  18.4   6.9
+ grep SUM
+ iperf -c ping.online.net -i 1 -P 10 -f M -t 5
[SUM]  0.0- 1.0 sec  7.50 MBytes  7.50 MBytes/sec
[SUM]  1.0- 2.0 sec  5.00 MBytes  5.00 MBytes/sec
[SUM]  2.0- 3.0 sec  4.25 MBytes  4.25 MBytes/sec
[SUM]  3.0- 4.0 sec  4.62 MBytes  4.62 MBytes/sec
[SUM]  4.0- 5.0 sec  4.62 MBytes  4.62 MBytes/sec
[SUM]  0.0- 5.5 sec  27.2 MBytes  4.91 MBytes/sec
+ dd if=/dev/zero of=/root/test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 22.7015 s, 47.3 MB/s


==============================================================================================================
Sat Aug  8 03:59:01 GMT-3 2015
+ mtr -rwc10 decix.de
Start: Sat Aug  8 03:59:01 2015
HOST: XXXXXXXXXXXXXXX.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                       0.0%    10    8.1   6.3   2.0   9.1   2.0
  2.|-- 82.211.0.1                          0.0%    10   12.4  13.1   8.7  21.8   3.5
  3.|-- fra3.xe-0-0-0.accelerated.de        0.0%    10   14.3  13.8   9.1  17.8   3.0
  4.|-- sgw2-te-0-0-2-3-ixp.fra.de-cix.net  0.0%    10    9.6  11.8   8.9  17.8   2.9
  5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
+ mtr -rwc10 4.2.2.2
Start: Sat Aug  8 03:59:12 2015
HOST: XXXXXXXXXXXXXXX.de              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- XX.XX.XX.XX                             0.0%    10    0.1   5.5   0.1  10.2   3.5
  2.|-- 82.211.0.1                                0.0%    10    0.3  10.2   0.2  15.3   5.4
  3.|-- xe-4-2-1-311.edge4.Frankfurt1.Level3.net  0.0%    10    0.6  13.8   0.6  36.0  11.3
  4.|-- ae-1-60.edge5.Frankfurt1.Level3.net      10.0%    10    0.7  11.3   0.7  19.0   6.7
  5.|-- b.resolvers.Level3.net                    0.0%    10    0.7  10.2   0.6  16.0   5.6
+ grep SUM
+ iperf -c ping.online.net -i 1 -P 10 -f M -t 5
[SUM]  0.0- 1.0 sec  14.6 MBytes  14.6 MBytes/sec
[SUM]  1.0- 2.0 sec  11.1 MBytes  11.1 MBytes/sec
[SUM]  2.0- 3.0 sec  11.1 MBytes  11.1 MBytes/sec
[SUM]  3.0- 4.0 sec  5.75 MBytes  5.75 MBytes/sec
[SUM]  4.0- 5.0 sec  4.88 MBytes  4.88 MBytes/sec
[SUM]  0.0- 5.6 sec  48.8 MBytes  8.74 MBytes/sec
+ dd if=/dev/zero of=/root/test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 27.411 s, 39.2 MB/s

Comments

  • How the fuck should we know? Contact the support of the provider!?

  • @nexusrain said:
    How the fuck should we know? Contact the support of the provider!?

    Wait what? This isn't support? They told me to contact LowEndHelpDesk first hand before anything. Did no rite place? /troll

  • SplitIceSplitIce Member, Host Rep

    Did you only see this via DE-CIX?

    We had a customer report some packet loss via DE-CIX last night (but we couldnt replicate it ourselves via any of our monitoring servers, and our upstream couldnt relate it to anything on their network).

    But @nexusrain is right, you should be contacting your host with these MTRs.

  • andy_randy_r Member
    edited August 2015

    It is almost constant. Here are the latest results:

    Sun Aug  9 09:59:01 GMT-3 2015
    + mtr -rwc10 decix.de
    Start: Sun Aug  9 09:59:01 2015
    HOST: XXXXXXXXXX.ip-projects.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX.                       0.0%    10    2.9   4.3   0.1   9.3   3.2
      2.|-- 82.211.0.1                          0.0%    10   17.6   9.3   0.2  17.6   5.5
      3.|-- fra3.xe-0-0-0.accelerated.de        0.0%    10   12.7  11.9   0.4  20.4   5.5
      4.|-- sgw2-te-0-0-2-3-ixp.fra.de-cix.net  0.0%    10   14.8  10.8   1.3  17.4   5.6
      5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
    + mtr -rwc10 4.2.2.2
    Start: Sun Aug  9 09:59:11 2015
    HOST: XXXXXX.ip-projects.de              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX                             0.0%    10    1.7   4.7   1.7   9.0   2.4
      2.|-- 82.211.0.1                                0.0%    10    9.9  12.5   4.8  35.6   8.5
      3.|-- xe-4-2-1-311.edge4.Frankfurt1.Level3.net  0.0%    10    6.9  15.8   6.9  51.5  12.8
      4.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- b.resolvers.Level3.net                    0.0%    10   12.6  12.8   7.8  15.9   2.4
    + grep SUM
    + iperf -c ping.online.net -i 1 -P 10 -f M -t 5
    [SUM]  0.0- 1.0 sec  7.12 MBytes  7.12 MBytes/sec
    [SUM]  1.0- 2.0 sec  4.50 MBytes  4.50 MBytes/sec
    [SUM]  2.0- 3.0 sec  4.75 MBytes  4.75 MBytes/sec
    [SUM]  3.0- 4.0 sec  4.62 MBytes  4.62 MBytes/sec
    [SUM]  4.0- 5.0 sec  4.38 MBytes  4.38 MBytes/sec
    [SUM]  0.0- 5.6 sec  26.6 MBytes  4.79 MBytes/sec
    + dd if=/dev/zero of=/root/test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 6.31206 s, 170 MB/s
    

    That was the first thing I did, asking support. I started tracert-ing de-cix because asked by the support. Before i was testing other destinations with the same results.

    + date
    Fri Aug  7 12:59:01 GMT-3 2015
    + mtr -rwc10 ping.online.net
    Start: Fri Aug  7 12:59:01 2015
    HOST: XXXXXXXXXX.ip-projects.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX                       0.0%    10    8.7   7.6   3.2  10.7   2.2
      2.|-- 82.211.0.1                          0.0%    10   16.3  16.6   8.0  46.2  11.1
      3.|-- ae0-318.fra61.ip4.gtt.net           0.0%    10   12.7  14.2   8.9  20.3   3.4
      4.|-- et-4-1-0.par70.ip4.gtt.net          0.0%    10   20.4  25.4  20.3  34.0   5.0
      5.|-- online-gw.ip4.gtt.net               0.0%    10   23.7  24.9  19.4  30.1   4.2
      6.|-- 45x-s44-2-a9k2.dc3.poneytelecom.eu  0.0%    10   24.4  25.2  20.7  34.8   4.2
      7.|-- ping.online.net                     0.0%    10   20.3  22.0  17.7  26.7   2.9
    + grep SUM
    + iperf -c ping.online.net -i 1 -P 10 -f M -t 5
    [SUM]  0.0- 1.0 sec  7.50 MBytes  7.50 MBytes/sec
    [SUM]  1.0- 2.0 sec  5.00 MBytes  5.00 MBytes/sec
    [SUM]  2.0- 3.0 sec  3.75 MBytes  3.75 MBytes/sec
    [SUM]  3.0- 4.0 sec  4.12 MBytes  4.12 MBytes/sec
    [SUM]  4.0- 5.0 sec  4.75 MBytes  4.75 MBytes/sec
    [SUM]  0.0- 5.5 sec  26.4 MBytes  4.77 MBytes/sec
    + mtr -rwc10 nyse.com
    Start: Fri Aug  7 12:59:17 2015
    HOST: XXXXXXXXX.ip-projects.de  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX                 0.0%    10    3.1   2.6   0.1   8.4   3.1
      2.|-- 82.211.0.1                    0.0%    10   16.0  12.9   0.3  39.9  12.4
      3.|-- fra3.xe-0-0-0.accelerated.de  0.0%    10   11.2   8.5   0.4  36.0  11.2
      4.|-- decix.accelerated.de          0.0%    10   12.9   7.4   0.4  15.6   6.8
      5.|-- cloudflare.fra.ecix.net       0.0%    10    8.4   5.7   0.7  17.4   7.0
      6.|-- 104.16.1.25                   0.0%    10   18.2   6.5   0.6  18.2   6.6
    

    Chatting with support the third day about the problem, provided all logs, started tracerouting de-cix as asked, at the end was told it is OK, getting much more stable results on vpns from 23Media and filoo: 1-2ms to de-cix and 10ms less behind it to the same destinations and 10MB/s throughput. ip-projects vpn seems hugely overloaded (or misconfigured), but their support says results are normal, I am not sure

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited August 2015

    You can't compare ip-projects or one of their resellers to accelerated directly, I had similar issues a while ago (2 years) and ended up speaking to accelerated directly they were actually pretty annoyed that the reseller of a customer was suggesting there network setup was the same as using accelerated directly, they gave me a free test server for a week to prove the point and the difference was insane.

    That said.. What nic type are you using, have you installed PV on HVM drivers and has your host enabled that on your HVM vps and do you see the same latency results over tcp, what is the host node connected at?

  • andy_randy_r Member
    edited August 2015

    ip-project uses XEN full virtualizatoon

    root:~# lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                1
    On-line CPU(s) list:   0
    Thread(s) per core:    1
    Core(s) per socket:    1
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 62
    Model name:            Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
    Stepping:              4
    CPU MHz:               2100.038
    BogoMIPS:              4200.07
    Hypervisor vendor:     Xen
    Virtualization type:   full
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              15360K
    NUMA node0 CPU(s):     0
    

    As such, I thought there were no need to install drivers.
    Since I get sometimes normal results like these

    Mon Aug 10 08:59:01 GMT-3 2015
    + mtr -rwc10 decix.de
    Start: Mon Aug 10 08:59:02 2015
    HOST: XXXXXXXXX.ip-projects.de        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX                       0.0%    10    0.2   0.1   0.1   0.2   0.0
      2.|-- 82.211.0.1                          0.0%    10    0.3   1.5   0.2  11.3   3.4
      3.|-- fra3.xe-0-0-0.accelerated.de        0.0%    10    0.4   0.4   0.4   0.4   0.0
      4.|-- sgw2-te-0-0-2-3-ixp.fra.de-cix.net  0.0%    10    1.4   1.3   1.2   1.5   0.0
      5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
    + mtr -rwc10 4.2.2.2
    Start: Mon Aug 10 08:59:12 2015
    HOST: XXXXXXXXX.ip-projects.de              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- XX.XX.XX.XX                             0.0%    10    0.1   0.1   0.1   0.2   0.0
      2.|-- 82.211.0.1                                0.0%    10    0.3   3.9   0.2  20.9   6.7
      3.|-- xe-4-2-1-311.edge4.Frankfurt1.Level3.net  0.0%    10    0.6   0.7   0.6   1.0   0.0
      4.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- b.resolvers.Level3.net                    0.0%    10    0.6   1.3   0.6   6.9   1.9
    + grep SUM
    + iperf -c ping.online.net -i 1 -P 10 -f M -t 5
    [SUM]  0.0- 1.0 sec  13.8 MBytes  13.8 MBytes/sec
    [SUM]  1.0- 2.0 sec  11.2 MBytes  11.2 MBytes/sec
    [SUM]  2.0- 3.0 sec  11.4 MBytes  11.4 MBytes/sec
    [SUM]  3.0- 4.0 sec  11.5 MBytes  11.5 MBytes/sec
    [SUM]  0.0- 5.3 sec  60.4 MBytes  11.4 MBytes/sec
    + dd if=/dev/zero of=/root/test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 4.79334 s, 224 MB/s
    

    there should be no problem with setup, just a banal overload of the host machine most of the time.

  • IkoulaIkoula Member, Host Rep
    edited August 2015

    Hello,

    How is host load average ? Overloaded cpu might impact everything including disks and network performance.

    Edit : If the host load average is greater than 1 that will not change HW performance but that can impact vm performance so checking with #top command on the host is a good start i think.

  • AnthonySmithAnthonySmith Member, Patron Provider

    Well sorry but you are wrong.

  • I have no access to host console, I am vps customer. I have been testing three different vps providers last week, each has its ups and downs, will post results soon.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited August 2015

    That does not prevent you from checking your nic type and block drivers etc, in solusvm and other panels I have used its fairly standard to be able to change or update this as a customer.

  • andy_randy_r Member
    edited August 2015

    It does not make sense either.
    Anyway I am done with testing. Here are my observation about three vps providers tested:

    Rootwelt (23Media)
    Pros: Good uplink. The network showed best results for my location and was stable. They say they put no limit on traffic so I was afraid that users with large usage would be attracted, but in reality I did not noticed any problems with the network.
    Cons: One and huge: distros. OS images they provide are modified. In debian there are 100kb of junk in ENV which you see as output of “set” command. They assure you that this is “clean minimal installation”, but in fact tampered even with aptitude

    root@vmXXXX:~# apt-get update
    Get:1 http://security.debian.org wheezy/updates Release.gpg [1,554 B]
    Get:2 http://security.debian.org wheezy/updates Release [102 kB]
    Get:3 http://security.debian.org wheezy/updates/main amd64 Packages [317 kB]
    Get:4 http://security.debian.org wheezy/updates/main Translation-en [181 kB]
    Hit http://ftp.us.debian.org wheezy Release.gpg
    Hit http://ftp.us.debian.org wheezy Release
    Hit http://ftp.us.debian.org wheezy/main amd64 Packages
    Hit http://ftp.us.debian.org wheezy/main Translation-en
    Fetched 602 kB in 0s (1,115 kB/s)
    Reading package lists... Done
    W: There is no public key available for the following key IDs:
    9D6D8F6BC857C906
    W: There is no public key available for the following key IDs:
    7638D0442B90D010
    root@vmXXXX:~#
    

    and do not allow you to make clean install from official distribution, when asked. not good

    Ip-projects (Accelerated)
    Pros: Good uplink. Clean minimal installation. XEN full virtualizaton. Seem to follow the standards.
    Cons: Host machine is largely oversold/overloaded and they won'tt admit it. Do not expect stable network performance from them, could see hdd rates go down to 30 MB/s more than once.

    Filoo (Filoo)
    Pros: Good uplink. Have seen no problem with the load of the server at all. Was able to attach installation image myself and install distro from scratch myself (image they provide was also modified). Console works with internet browser without java, which is a very very nice thing, provided that in emergency situation when you will need console, there might be no ssh client at hand.
    Cons: More expensive. Less traffic included in the price. Also I noticed with tcpdump the packets from other subnets coming to my interface, which does not hurt much of course but was kinda weird.

  • DewlanceVPSDewlanceVPS Member, Patron Provider
    edited August 2015

    Downgrade to lower version of Xen and then recheck.



    Make sure setup and configuration of Xen server is correct.
    I am using Xen and never face any issue but you can try to get a help from Xen community forum. They can give you suggestion.



    Most of LET users use KVM.

  • AnthonySmithAnthonySmith Member, Patron Provider

    Well as your refusing to look into your own servers setup for some reason and keep saying full virtual like that's a catch all I can only take your results with a pinch of salt.

  • The reasons were all given above, including the fact that connectivity problems are not constant, although periodic. When they coincides with other indicators in vps which suggest that the problem lie beyond only network drivers (such as low hdd rates) I can see no point in looking into them, but I am thankful for your suggestion.

  • One comment: the issue with network misconfiguration @Filoo seems to be a bigger problem. I can tcpdump udp and tcp packets addressed to other machines, even voip protocol included with passwords etc. Thats SERIOUS. And they have not been able to fix IT in more than two weeks. Very VERY bad :(

  • Who is curious: these are packets to/from addresses neither of which is mine.
    Well, is it any good???

    root@filoo:~# tcpdump port 5060 and host not XXX.my.ip.address and ip and udp -vv -nn
    
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    10:21:22.511772 IP (tos 0x0, ttl 55, id 23929, offset 0, flags [DF], proto UDP (17), length 437)
        64.91.228.60.5062 > 93.190.68.9.5060: [udp sum ok] SIP, length: 409
            OPTIONS sip:[email protected] SIP/2.0
            Via: SIP/2.0/UDP 64.91.228.60:5062;branch=z9hG4bK-1919513746;rport
            Content-Length: 0
            From: "sipvicious";tag=3564626534343039313363340133313634363035323633
            Accept: application/sdp
            User-Agent: friendly-scanner
            To: "sipvicious"
            Contact: sip:[email protected]:5062
            CSeq: 1 OPTIONS
            Call-ID: 77673589852159372802078
            Max-Forwards: 70
            
    10:21:22.618565 IP (tos 0x0, ttl 56, id 58757, offset 0, flags [DF], proto UDP (17), length 439)
        64.91.228.60.5062 > 93.190.68.29.5060: [udp sum ok] SIP, length: 411
            OPTIONS sip:[email protected] SIP/2.0
            Via: SIP/2.0/UDP 64.91.228.60:5062;branch=z9hG4bK-3052826983;rport
            Content-Length: 0
            From: "sipvicious";tag=3564626534343164313363340134323835303538323333
            Accept: application/sdp
            User-Agent: friendly-scanner
            To: "sipvicious"
            Contact: sip:[email protected]:5062
            CSeq: 1 OPTIONS
            Call-ID: 284910327985329607020499
            Max-Forwards: 70
    

    ^C

Sign In or Register to comment.