All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Weird routing Choopa NJ - BuyVM Buffalo
Any ideas what might be going on here?
Provider A at Choopa -> BuyVM Buffalo:
`
2 ethernet17-7-br1.pnj1.choopa.net (108.61.92.113) 81.373 ms 81.365 ms 81.362 ms
3 ae11.ar2.nyc3.us.nlayer.net (69.31.34.37) 2.398 ms 2.329 ms 2.297 ms
4 as2914.ae10.ar2.nyc3.us.nlayer.net (69.31.34.193) 1.627 ms 1.797 ms 1.769 ms
5 xe-2.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.9.178) 1.086 ms 1.069 ms xe-0.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.30) 1.064 ms
6 nyk-bb1-link.telia.net (213.155.131.136) 1.080 ms 1.079 ms 1.098 ms
7 buf-b1-link.telia.net (80.91.246.36) 10.399 ms 10.379 ms 10.353 ms
8 giglinx-ic-155660-buf-b1.c.telia.net (213.248.96.42) 24.228 ms 24.716 ms *
9 199.195.255.1 (199.195.255.1) 10.380 ms 10.398 ms 10.395 ms
--- 199.195.255.1 ping statistics ---
17 packets transmitted, 17 received, 0% packet loss, time 16025ms
rtt min/avg/max/mdev = 10.409/10.491/10.581/0.138 ms
`
Provider B at Choopa -> BuyVM Buffalo:
`
2 ethernet13-7-br1.pnj1.choopa.net (108.61.92.145) 6.932 ms 6.970 ms 6.946 ms
3 ae11.ar2.nyc3.us.nlayer.net (69.31.34.37) 3.010 ms 2.960 ms 2.933 ms
4 as2914.ae10.ar2.nyc3.us.nlayer.net (69.31.34.193) 1.589 ms 1.633 ms 1.659 ms
5 xe-1.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.154) 1.050 ms xe-0.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.30) 1.086 ms 1.078 ms
6 nyk-bb1-link.telia.net (213.155.135.18) 17.919 ms 8.460 ms 8.398 ms
7 buf-b1-link.telia.net (80.91.246.36) 10.386 ms 10.346 ms 10.309 ms
8 giglinx-ic-155660-buf-b1.c.telia.net (213.248.96.42) 15.040 ms 15.439 ms 15.905 ms
9 199.195.255.1 (199.195.255.1) 23.761 ms 23.863 ms 23.832 ms
--- 199.195.255.1 ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20025ms
rtt min/avg/max/mdev = 23.832/23.998/24.603/0.225 ms
`
Comments
Edit: I looked back at this and I am a bit confused, this looks like two different servers on the same router just different vlans? Are you sure this is different providers? If these are not on the same providers, I would say it seems like one node is at high bandwidth saturation thus is causing some latency.
Two different VPS providers yes, they share the same network (ReliableSite). That's what's interesting about the latency differences, there should be none!
I have tried with another VPS provider (same ReliableSite network, let's call it provider C) and still pings at ~24ms :S
Got more interesting:
From VPS Provider A:
PING 199.195.255.1 (199.195.255.1) 56(84) bytes of data.
64 bytes from 199.195.255.1: icmp_req=1 ttl=55 time=10.5 ms
64 bytes from 199.195.255.1: icmp_req=2 ttl=55 time=10.4 ms
64 bytes from 199.195.255.1: icmp_req=3 ttl=55 time=10.4 ms
^C
--- 199.195.255.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.444/10.487/10.520/0.031 ms
PING 199.195.255.20 (199.195.255.20) 56(84) bytes of data.
64 bytes from 199.195.255.20: icmp_req=1 ttl=49 time=23.3 ms
64 bytes from 199.195.255.20: icmp_req=2 ttl=49 time=23.3 ms
64 bytes from 199.195.255.20: icmp_req=3 ttl=49 time=23.3 ms
^C
--- 199.195.255.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 23.309/23.328/23.352/0.125 ms
From VPS Provider B:
PING 199.195.255.1 (199.195.255.1) 56(84) bytes of data.
64 bytes from 199.195.255.1: icmp_req=1 ttl=49 time=23.8 ms
64 bytes from 199.195.255.1: icmp_req=2 ttl=49 time=23.9 ms
64 bytes from 199.195.255.1: icmp_req=3 ttl=49 time=23.9 ms
64 bytes from 199.195.255.1: icmp_req=4 ttl=49 time=23.9 ms
^C
--- 199.195.255.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 23.870/23.918/23.938/0.027 ms
PING 199.195.255.20 (199.195.255.20) 56(84) bytes of data.
64 bytes from 199.195.255.20: icmp_req=1 ttl=54 time=10.5 ms
64 bytes from 199.195.255.20: icmp_req=2 ttl=54 time=10.6 ms
64 bytes from 199.195.255.20: icmp_req=3 ttl=54 time=11.0 ms
64 bytes from 199.195.255.20: icmp_req=4 ttl=54 time=10.5 ms
^C
--- 199.195.255.20 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 10.557/10.697/11.044/0.225 ms
Maybe http://www.lowendtalk.com/discussion/comment/205798
As the providers are likely in different ranges i bet it is a return path issue back to choopa crom colocrossing. What does an mtr going the other direction look like?
I remember that, but in this case the routes are pretty similar :S
Unfortunately I don't have a Buffalo node to try on. I did check using the TeliaSonera LG but they were identical, and ~10ms both
@miguelq inbound is only 1/2 the equation. BGP route selection is prefix based so different origin ips will possibly have different return paths. This is one of the faults of natural bgp path sepection.
@miguelq the telia lg would be assuming your traffic left buyvm/cc on telia. Any other path would not be representative in that case.
@MiguelQ I can provide a Buffalo node for testing, if you want.
I understand, the point was to eliminate telia-path issues since it arrived to BuyVM via Telia. Look at this, from the same host to different IP in same range of BuyVM:
root@db1:~# traceroute 199.195.255.1
traceroute to 199.195.255.1 (199.195.255.1), 30 hops max, 60 byte packets
2 ethernet17-7-br1.pnj1.choopa.net (108.61.92.113) 10.075 ms 10.118 ms 10.132 ms
3 ae11.ar2.nyc3.us.nlayer.net (69.31.34.37) 4.480 ms 4.437 ms 2.154 ms
4 as2914.ae10.ar2.nyc3.us.nlayer.net (69.31.34.193) 1.530 ms 1.696 ms 1.677 ms
5 xe-0.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.30) 1.114 ms xe-2.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.9.178) 1.038 ms xe-0.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.30) 1.010 ms
6 nyk-bb1-link.telia.net (213.155.131.136) 1.069 ms 1.090 ms 1.102 ms
7 buf-b1-link.telia.net (80.91.246.36) 10.354 ms 10.370 ms 10.337 ms
8 giglinx-ic-155660-buf-b1.c.telia.net (213.248.96.42) 24.252 ms 24.883 ms 25.264 ms
9 199.195.255.1 (199.195.255.1) 10.457 ms 10.429 ms 10.560 ms
root@db1:~# traceroute 199.195.255.20
traceroute to 199.195.255.20 (199.195.255.20), 30 hops max, 60 byte packets
2 ethernet17-7-br1.pnj1.choopa.net (108.61.92.113) 0.273 ms 0.316 ms 0.331 ms
3 ae11.ar2.nyc3.us.nlayer.net (69.31.34.37) 2.330 ms 2.282 ms 2.832 ms
4 as2914.ae10.ar2.nyc3.us.nlayer.net (69.31.34.193) 1.468 ms 1.667 ms 1.643 ms
5 xe-2.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.9.178) 1.063 ms 1.040 ms xe-1.teliasonera.nycmny01.us.bb.gin.ntt.net (129.250.8.154) 1.009 ms
6 nyk-bb1-link.telia.net (213.155.130.246) 1.150 ms 1.132 ms 1.152 ms
7 buf-b1-link.telia.net (80.91.246.36) 10.347 ms 10.371 ms 10.379 ms
8 giglinx-ic-155660-buf-b1.c.telia.net (213.248.96.42) 24.339 ms 24.808 ms 25.265 ms
9 * * *
10 199.195.255.20 (199.195.255.20) 23.412 ms 23.381 ms 23.345 ms
--- 199.195.255.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 10.401/10.480/10.550/0.117 ms
--- 199.195.255.20 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 23.336/23.437/23.537/0.209 ms
Sent you a PM. Thank you!
You can also try http://ny.lg.cloudshards.net --> Buffalo, Colocrossing for testing.
Will do, thanks!
CC has been doing some route adjustments as far as I know so it's possible one of the paths is taking Cogent back.
I'll get an lg setup tonight.
Francisco
Thanks to @concerto49:
To Provider A (SSDVirt):
HOST: ny Loss% Snt Last Avg Best Wrst StDev 1. csny001.cloudshards.com 0.0% 10 0.0 0.0 0.0 0.1 0.0 2. gw-buf.ny.routers.cloudshards.net 20.0% 10 0.5 0.5 0.5 0.6 0.0 3. te7-4.ccr01.buf02.atlas.cogentco.com 0.0% 10 1.9 29.5 0.3 157.2 61.4 4. te7-8.ccr02.cle04.atlas.cogentco.com 0.0% 10 8.7 22.5 5.4 173.2 52.9 5. te0-5-0-6.ccr21.ord01.atlas.cogentco.com 0.0% 10 13.8 13.8 13.7 13.9 0.1 6. te0-1-0-1.ccr22.ord03.atlas.cogentco.com 0.0% 10 13.4 13.5 13.4 13.6 0.1 7. sprint.ord03.atlas.cogentco.com 0.0% 10 13.5 13.6 13.4 14.1 0.2 8. 144.232.1.103 0.0% 10 15.0 25.8 15.0 83.2 21.6 9. sl-crs1-chi-0-8-0-2.sprintlink.net 0.0% 10 21.3 24.7 21.1 52.1 9.7 10. 144.232.5.217 0.0% 10 23.9 24.0 23.4 24.8 0.4 11. 144.232.4.85 0.0% 10 23.2 23.3 23.1 23.5 0.1 12. 144.223.37.122 0.0% 10 26.5 24.8 23.5 26.6 1.1 13. as20473.ae7.ar1.nyc3.us.nlayer.net 0.0% 10 23.9 24.3 23.9 27.1 1.0 14. ve20-br1.pnj1.choopa.net 0.0% 10 34.6 27.2 23.7 35.4 5.4 15. 108.61.92.114.choopa.net 0.0% 10 24.1 24.1 24.0 24.1 0.0 16. -61-49-168.static.ssdvirt.net 0.0% 10 23.9 23.9 23.8 24.0 0.1To Provider B (ArmorShark):
HOST: ny Loss% Snt Last Avg Best Wrst StDev 1. csny001.cloudshards.com 0.0% 10 0.0 0.0 0.0 0.1 0.0 2. gw-buf.ny.routers.cloudshards.net 0.0% 10 0.5 0.5 0.5 0.5 0.0 3. buf-b1-link.telia.net 0.0% 10 0.2 0.2 0.2 0.2 0.0 4. nyk-bb1-link.telia.net 0.0% 10 9.4 13.4 9.4 48.9 12.5 5. nyk-b5-link.telia.net 0.0% 10 9.5 12.0 9.5 34.7 8.0 6. xe-10-1-0.edge1.NewYork1.Level3.net 0.0% 10 9.6 10.9 9.5 23.1 4.3 7. ae-3-80.edge3.NewYork1.Level3.net 0.0% 10 9.5 9.6 9.5 9.6 0.0 8. CHOOPA-LLC.edge3.NewYork1.Level3.net 0.0% 10 11.7 15.7 11.7 21.9 3.0 9. ve67-br1.pnj1.choopa.net 0.0% 10 11.1 13.1 11.1 30.4 6.1 10. 108.61.92.146.choopa.net 0.0% 10 11.1 11.1 11.0 11.2 0.1 11. hosted-by.armorshark.com 0.0% 10 11.0 11.0 11.0 11.2 0.1So the difference between providers are a cause of the return path (Cogent-Chicago).
@RyanD But what could explain the difference from the same host to IPs in the same range (199.195.255.1, 199.195.255.20)?
Because your return-path to one provider is cogent and your other provider is via telia.
Your source IP's prefix determines the return path either based upon forced route selection via IP prefix list or destination AS numbers. In this case it seems unlikely to be an as-path force as they are both originating from Choopa's AS so I would chalk that path selection up to natrual BGP's decision making. I doubt CC would be making manual prefix list path preferences as that would be a configuration nightmare to maintain
I just tested a number of our IPs and it's quite weird. IPs ending with a odd number seem to take Telia and then Level3 while IPs ending with a even number take Cogent. I'm do not know why this is occurring but will contact our provider regarding this.
@MiguelQ, could you test an IP that ends with a even number at your other provider.
@John if that is a case that you have been able to reliably duplicate then that indicates that upstream they are using an Odd/Even policy-based routing configuration to set ip next-hop or bgp route preference for odd IP based destinations to one carrier and even to another.
@john I got same result as you. @RyanD which upstream would be using such policy in this case? CC or ReliableSite?
In this case, it would be CC
What would be the point of this?
@Spencer it's a form of load balancing.
@RyanD, is there anything we can do besides contact CC?
@MiguelQ, if the routing is a problem for you, contact us and we'll issue you an odd numbered IP.
@John if it's their routing policy to route like that you can just mess with the odd/even destinations if you know that is how they are setup. Otherwise, yeah you'll need to ask them to make changes
It's time for Colocrossing to get a http://www.noction.com/
@RyanD I know you guys have an Internap FCP
@concerto49
Noction is a nice product but not quite mature enough. They are slowly adding features that get it comparable with the Internap product. They do present a pretty strong value position compared to Internap
We've been running the FCP platform for about 4 years now in Atlanta, 2 in Phoenix, and a few months in our LA location. They perform great. However, their cost is well... staggering
No problem on our side John, I was considering getting some KVM in Buffalo and needed the low latency, but I think I'd better wait until these routing issues are resolved/explained/sorted out.