All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
BuyVM routing issues at EGI/Coresite
quirkyquark
Member
Preface: I'm a BuyVM customer, and while this routing/bandwidth problem isn't affecting my "play server" yet, I post because it's affecting others and is an interesting conundrum from a network engineering perspective. My gut feeling is to chalk this down to a router misconfiguration someplace -- trolls please take conspiracy theories elsewhere.
Using curl instead of wget since it does tcp-nodelay. Two test files: BuyVM and Lightwave, both VPS providers colocated with EGI at the Coresite DC in San Jose, CA, and using the same mix of carriers/bandwidth (NLayer, HE.Net, and Bandcon/Highwinds). Tests run sequentially.
Test from Kansas City, USA (Wholesale/Datashack DC). First is always Lightwave, second is BuyVM. Average speeds are bolded.
root@ubs:~# date; curl -otest --tcp-nodelay http://ping.lightwave.net/test100MB.bin; date; curl -otest --tcp-nodelay http://buyvm.net/100mb.test Sat Mar 31 21:54:21 MSK 2012 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 100 100M 0 0 **35.9M** 0 0:00:02 0:00:02 --:--:-- 41.1M Sat Mar 31 21:54:24 MSK 2012 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 100 100M 0 0 **2466k** 0 0:00:41 0:00:41 --:--:-- 3410k
BuyVM should be as insanely fast as Lightwave because this is being router over Hurricane Electric (he.net) from end to end.
Traceroutes: I'll post this one here, and link to the remaining on pastebin, because of character limits and because they are identical except for the one extra hop BuyVM always requires through what appears to be an EGI router:
Sat Mar 31 21:55:28 MSK 2012 traceroute to ping.lightwave.net (199.68.196.41), 30 hops max, 60 byte packets 1 173.208.233.66 (173.208.233.66) 0.026 ms 0.005 ms 0.005 ms 2 63.141.255.129 (63.141.255.129) 0.215 ms 0.301 ms 0.384 ms 3 69.30.209.221 (69.30.209.221) 0.355 ms 0.415 ms 0.473 ms 4 10gigabitethernet1-3.core1.mci2.he.net (216.66.78.89) 0.316 ms 0.355 ms 0.415 ms 5 10gigabitethernet1-1.core1.mci3.he.net (184.105.213.38) 4.705 ms 4.782 ms 4.834 ms 6 10gigabitethernet5-1.core1.den1.he.net (184.105.222.21) 21.812 ms 14.759 ms 14.809 ms 7 10gigabitethernet11-4.core1.sjc2.he.net (184.105.213.105) 41.264 ms 39.121 ms 39.232 ms 8 10gigabitethernet1-4.core1.sjc1.he.net (72.52.92.117) 46.473 ms 41.272 ms 46.585 ms 9 energy-group-networks-llc.10gigabitethernet1-3.core1.sjc1.he.net (64.71.150.22) 39.302 ms 39.378 ms 39.485 ms 10 Edge1.SJC1.Lightwave.Net (199.68.196.41) 39.311 ms 39.253 ms 39.285 ms Sat Mar 31 21:55:29 MSK 2012 traceroute to buyvm.net (205.185.112.61), 30 hops max, 60 byte packets 1 173.208.233.66 (173.208.233.66) 0.025 ms 0.005 ms 0.005 ms 2 63.141.255.129 (63.141.255.129) 0.211 ms 0.288 ms 0.386 ms 3 69.30.209.221 (69.30.209.221) 0.240 ms 0.341 ms 0.391 ms 4 10gigabitethernet1-3.core1.mci2.he.net (216.66.78.89) 0.355 ms 0.405 ms 0.474 ms 5 10gigabitethernet1-1.core1.mci3.he.net (184.105.213.38) 0.351 ms 0.412 ms 0.464 ms 6 10gigabitethernet5-1.core1.den1.he.net (184.105.222.21) 12.057 ms 12.109 ms 12.105 ms 7 10gigabitethernet11-4.core1.sjc2.he.net (184.105.213.105) 49.578 ms 49.753 ms 46.808 ms 8 10gigabitethernet1-4.core1.sjc1.he.net (72.52.92.117) 39.199 ms 39.299 ms 41.437 ms 9 energy-group-networks-llc.10gigabitethernet1-3.core1.sjc1.he.net (64.71.150.22) 49.184 ms 49.284 ms 49.219 ms 10 **173.245.86.18** (173.245.86.18) 39.273 ms 39.219 ms 39.194 ms 11 jael.billing.frantech.ca (205.185.112.61) 39.528 ms 39.546 ms 39.564 ms
The bolded IP (173.245.86.18) is always the last hop for BuyVM, and is the only difference between it and Lightwave.
From San Diego, CA (AIS DC):
Sat Mar 31 17:13:23 PDT 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **11.0M** 0 0:00:09 0:00:09 --:--:-- 9.7M
Sat Mar 31 17:13:32 PDT 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **3837k** 0 0:00:26 0:00:26 --:--:-- 3937k
From Dusseldorf (Misterhost.de):
Sun Apr 1 04:29:49 MSK 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **4483k** 0 0:00:22 0:00:22 --:--:-- 7935k
Sun Apr 1 04:30:12 MSK 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **654k** 0 0:02:36 0:02:36 --:--:-- 1001k
As far East as I can get, from Tallinn, Estonia (Torqhost):
Sun Apr 1 04:37:29 MSK 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **5599k** 0 0:00:18 0:00:18 --:--:-- 7323k
Sun Apr 1 04:37:48 MSK 2012
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 **549k** 0 0:03:06 0:03:06 --:--:-- 357k
And this is the shocker -- from a BuyVM VPS:
root@bvm128:/tmp# date; curl -otest --tcp-nodelay http://ping.lightwave.net/test100MB.bin; date; curl -otest --tcp-nodelay http://buyvm.net/100mb.test Sun Apr 1 04:42:32 MSK 2012 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 100 100M 0 0 **41.5M** 0 0:00:02 0:00:02 --:--:-- 41.6M Sun Apr 1 04:42:35 MSK 2012 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 100 100M 0 0 **19.5M** 0 0:00:05 0:00:05 --:--:-- 20.1M
Traceroute:
root@bvm128:/tmp# date;traceroute ping.lightwave.net;date;traceroute buyvm.net Sun Apr 1 04:44:00 MSK 2012 traceroute to ping.lightwave.net (199.68.196.41), 30 hops max, 60 byte packets 1 205.185.112.148 (205.185.112.148) 0.062 ms 0.016 ms 0.013 ms 2 173.245.86.17 (173.245.86.17) 10.506 ms 10.417 ms 10.490 ms 3 Edge1.SJC1.Lightwave.Net (199.68.196.41) 0.913 ms 0.827 ms 0.777 ms Sun Apr 1 04:44:20 MSK 2012 traceroute to buyvm.net (205.185.112.61), 30 hops max, 60 byte packets 1 205.185.112.148 (205.185.112.148) 0.128 ms 0.024 ms 0.020 ms 2 jael.billing.frantech.ca (205.185.112.61) 0.519 ms 1.153 ms 1.054 ms
Note that a mysterious EGI router pops up in the trace to lightwave.net here, but BuyVM is still slower.
Why is the pony limping so badly? An unknown stud from the next-door stable is beating it mercilessly, even on its own private racetrack!!! For shame, sirs.... @Aldyric , @Francisco .... get him in shape!!! 
(happy to run any tests needed to troubleshoot; additional DC options are Peer1@LAX, ServerCentral@Chicago, Prometeus@Milan and Snel@Amsterdam.
Comments
Just want to add some additional information about this, from GNAX Atlanta, I can get 10mb/s constantly from BuyVM, through nLayer, and about 20mb/s from Infinitie.net, through he.net, I guess this has something to do with the choice of network carriers.
Also from my ISP, I have three different routes to BuyVM, one's he.net, one's globalcrossing and the other is nLayer. Among those routes, globalcrossing is the best, at 160ms ~ 190ms, and then he.net, at 200ms ~ 220ms, the last is nLayer, which went bad recently, from 180ms ~ 500ms.
He.net was once terrible but now it became more stable for me now as many escaped from them, and the opposite goes to nLayer, seems to be over saturated.
The lag at the hop is just an RDNS thing
I'll see if Egi actually delegated it to us and if they did i'll set it to something funny.
EDIT - Remember, lightwave doesn't run their own BGP so you'll always have a HOP from us. We can make the hop disappear easily enough though.
Francisco
@quirkyquark This is a nice post and all, but have you created a ticket with BuyVM?
Posting on a forum may feel like a good solution to a problem, but you really should talk to the provider first.
He's likely trying to get a feel for what's up just like we're trying to figure out what's being retarded on a lot of this.
We've already had a chat with egi and they state there's no limits/etc on their end but I dunno.
If you can give me a bit i'm going to throw a file up on node58 which has a probable fix that I can work from.
That, or you maybe throw me a shell on one of these boxes?
Francisco
The box this is on is actually a KVM on a really crappy CPU/setup so i'm not surprised on this. I'll be moving it to the mirrors ATOM in a few and put a new URL.
Francisco
See the preface to my post: this is NOT a problem for ME, I could care less; but the pattern is fascinating to me as someone with a decent theoretical/practical knowledge of networking. I'm only bothering with this since I recently joined the BuyVM community.
Happy to, I pm'd you with two -- let me know if you need others.
I've sent a bunch more logs to egi so we'll see what they say.
I've found another client on the same router as us that has the same issue. @mitgib threw me a temp shell to run some SCP's for and I get identical speeds and I simply can't push higher. Said client also reported that his BW graphs took a major dive and he can't break 100Mbit outbound on a full gig port. They did do router upgrades last Saturday so I'm hoping it's just a bad line card/fibre, etc.
Fingers crossed.
Francisco
If you like, since some of my vps were used for test I can give you access to one for testing. Please let me know.
What makes you think I will ever remember to remove it, or even want to?
Oh, how sweet!
Awww
xoxo
Gawd, y'all are too much. All I was saying is I don't care about the shell, keep it, it's good for testing stuff
HAHAHA I really laughed at this so hard
bump.
Could I please get some reports back on if this is now resolved? Our new switches were installed last night and i've personally seen a very favourable increase in speed (from 300K per stream to 5MB/sec per stream to my house).
http://www.speedtest.net/result/1901021197.png - I have a 50/3 line at home.
Previous to the updates it was sitting around 4 - 5Mbit/sec on the inbound.
http://speedtest.buyvm.net as well.
Much thanks,
Francisco
@Francisco
SecureDragon (Tampa, FL):
Kiloserve (Los Angles, CA)
VMHosts.co.uk (UK):
InceptionHosting (NL):
@dmmcintyre3
Compare it to the lightwave tests please?
Thank you,
Francisco
Kiloserve:
SecureDragon
VMHosts.co.uk
InceptionHosting:
From FiberVolt Chicago:
LW:
wget -O /dev/null http://ping.lightwave.net/test100MB.bin
--2012-04-18 20:17:06-- http://ping.lightwave.net/test100MB.bin
Resolving ping.lightwave.net... 199.68.196.41
Connecting to ping.lightwave.net|199.68.196.41|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'
100%[======================================>] 104,857,600 27.9M/s in 4.3s
2012-04-18 20:17:10 (23.4 MB/s) - `/dev/null' saved [104857600/104857600]
BUYVM:
wget speedtest.buyvm.net/100mb.test -O /dev/null
--2012-04-18 20:18:32-- http://speedtest.buyvm.net/100mb.test
Resolving speedtest.buyvm.net... 205.185.112.31
Connecting to speedtest.buyvm.net|205.185.112.31|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'
100%[======================================>] 104,857,600 13.4M/s in 8.0s
2012-04-18 20:18:40 (12.5 MB/s) - `/dev/null' saved [104857600/104857600]
@dmmcintyre3 - danke
Dammit @kujoe, be even for once
The rest looks pretty well spot on with a little sway at times.
I'm happy, we'll see what peoples VPN, etc, speeds report back.
Thank you,
Francisco
@JoeMerit
Is there a route difference? What does a trace show?
Francisco
From FiberVolt Chicago:
BUYVM:
traceroute to speedtest.buyvm.net (205.185.112.31), 30 hops max, 60 byte packets
1 host.colocrossing.com (108.174.54.130) 0.015 ms 0.007 ms 0.006 ms
2 host.colocrossing.com (66.225.231.49) 5.841 ms 6.089 ms 6.301 ms
3 po4.ar2.ord6.us.scnet.net (204.93.192.189) 0.435 ms 0.578 ms 0.719 ms
4 62.po4.ar2.ord1.us.scnet.net (75.102.3.229) 19.368 ms 19.536 ms 19.673 ms
5 ae3-81.cr2.ord1.us.nlayer.net (69.31.111.9) 1.262 ms 1.250 ms 1.241 ms
6 xe-4-0-0.cr1.sjc1.us.nlayer.net (69.22.142.7) 52.541 ms 52.590 ms 52.578 ms
7 ae1-40g.ar2.sjc1.us.nlayer.net (69.22.143.118) 53.626 ms 55.962 ms 53.639 ms
8 as18779.xe-4-0-4.ar2.sjc1.us.nlayer.net (69.22.153.94) 59.892 ms 60.057 ms 61.031 ms
9 173.245.86.18 (173.245.86.18) 52.817 ms 52.837 ms 52.933 ms
10 mirrors.buyvm.net (205.185.112.31) 53.385 ms 52.929 ms 53.417 ms
LW:
traceroute to ping.lightwave.net (199.68.196.41), 30 hops max, 60 byte packets
1 host.colocrossing.com (108.174.54.130) 0.026 ms 0.007 ms 0.006 ms
2 host.colocrossing.com (66.225.231.49) 0.871 ms 1.152 ms 1.311 ms
3 po4.ar1.ord6.us.scnet.net (204.93.192.185) 0.672 ms 0.814 ms 0.936 ms
4 61.po4.ar1.ord1.us.scnet.net (75.102.3.225) 1.438 ms 1.548 ms 1.649 ms
5 ae0-81.cr1.ord1.us.nlayer.net (69.31.111.1) 1.136 ms 1.131 ms 1.123 ms
6 xe-0-0-0.cr1.slc1.us.nlayer.net (69.22.142.102) 37.333 ms 37.292 ms ae0-40g.cr2.ord1.us.nlayer.net (69.31.111.154) 1.364 ms
7 xe-4-0-0.cr1.sjc1.us.nlayer.net (69.22.142.7) 52.555 ms 52.603 ms xe-2-3-1.cr1.sfo1.us.nlayer.net (69.22.142.97) 54.301 ms
8 ae3-20g.cr1.sjc1.us.nlayer.net (69.22.143.161) 54.093 ms ae1-40g.ar2.sjc1.us.nlayer.net (69.22.143.118) 54.017 ms ae3-20g.cr1.sjc1.us.nlayer.net (69.22.143.161) 54.066 ms
9 ae1-40g.ar2.sjc1.us.nlayer.net (69.22.143.118) 55.940 ms 55.057 ms 55.127 ms
10 as18779.xe-4-0-4.ar2.sjc1.us.nlayer.net (69.22.153.94) 60.848 ms 60.940 ms 60.991 ms
11 Edge1.SJC1.Lightwave.Net (199.68.196.41) 52.686 ms 52.727 ms 54.276 ms
It is a different route but I'll for sure get a shell outside us and see how things go.
Thanks
Francisco
With SSH tunnel to buy VPS:
All tests done from my home connection.
The file download form speedtest.buyvm.net seems fine but the speedtests still suck.
Offtopic: Doesn't posting images work anymore?
You need to wrap it in tags, it doesn't auto parse I don't think.
I'll have Anthony check all of the nodes entropy pools to see if there's anything else a miss.
The top wget, is that a big improvement from before? or you never tested, etc?
Francisco
Thanks, if i use the code in this comment it works but when i copy&paste it to the other one it doesn't o0
Thanks, i'm on node 58 if that helps.
Chicago/ServerCentral:
Scranton/Burstnet:
Kansas City/Datashack-WII
Estonia -- as far east as I can go --- 40+ Mbits/sec!
Netherlands/Snel
As far as I can tell...what problem?
Great job, Team Pony!
The only weird part was testing from @prometeus ... both buyvm/lightwave were crawling at ~200 kb/sec. As far as I can tell, the problem is the transatlantic Global Crossing link; Snel (via HE) and Torqhost (via EUNETIP) have no problems. So that may explain why some folks in Europe aren't seeing the improvements.
@gsrdgrdghd how does the speed.io image work exactly? Is that you testing through a VPN, or you testing directly from home?
@quirkyquark - Thanks for the tests
it's looking good.
Francisco
I think it works the same way as speedtest.net, via flash. I have set up putty for SSH tunneling and am using 127.0.0.1:1080 as the proxy in Chrome.
Anyway this is also interesting:
Speedtests using your 100mb file:
From misterhost.de:
357 KB/s
From edis.at (AT location):
4.17 MB/s
Speed difference seems to correlate with routing difference.
You can't beat Einstein...Cali is 200+ ms from Eastern Europe, so if you're using a VPN and browsing without something like FasterFox (multiple connections, HTTP/1.1 pipelining enabled), even loading "dense" websites may seem "slow".