New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
China Unicom Fiber Cuts
concerto49
Member
in General
Saw it on WHT http://www.webhostingtalk.com/showthread.php?t=1317094
Probably explains why Tinet is routing to Europe earlier and the issues faced. It's still ongoing.
Anyone else notice anything strange?
Comments
It says they were cut, I wonder if it was on purpose or an accident. If by cable line they mean internet back bone, that could a serious issue.
This doesn't explain why in recent 2 months, traffic come out from China telecom backbone are routed to Ireland first then routed to West Coast of the US. For Great Firewall, you never know what's gonna happen.
That's not a China Telecom problem? That's likely a USA carrier having a peering dispute with China Telecom. The provider should be able to fix this by re-routing it via another carrier. You shouldn't be seeing these problems.
I know this post is about China unicom. but China telecom has problem too.
For nearly all IPs from LA that are not marked as "China-optimized", when traffic come out from China telecom backbone, it's routed to Ireland then goes to West Coast. This is a known issue which has last for nearly 2 months. People who play with VPSs in China nearly all know this fact. I show you a few examples:
when I traceroute an LA IP from China telecom by using the trace tool from 17ce.com (this is a common tool for traceroute from Chinese IPs)
eg1:
8 202.97.58.122 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn China Telecom backbone network 0% 2.537ms 2.038ms 2.22ms
9 202.97.52.102 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn China Telecom backbone network 16.67% 313.696ms 313.628ms 313.66ms
10 213.200.79.237 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Ireland 0% 341.683ms 338.303ms 339.18ms
11 141.136.110.193 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Ireland 50% 416.113ms 415.447ms 415.867ms
12 199.229.230.18 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn United States 83.33% 405.257ms 405.257ms 405.257ms
13 96.44.180.42 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Los Angeles, California QuadraNet Networks
eg2:
8 202.97.53.226 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Backbone network of China Telecom Beijing international export 0% 2.181ms 1.746ms 1.899ms
9 202.97.52.102 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn China Telecom backbone network 0% 314.261ms 313.882ms 314.015ms
10 213.200.79.237 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Ireland 50% 484.216ms 482.651ms 483.181ms
11 141.136.110.193 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Ireland 33.33% 385.223ms 381.785ms 383.111ms
12 199.229.230.18 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn United States 16.67% 488.502ms 487.009ms 487.85ms
13 96.44.180.42 93.23.142.219.broad.bj.bj.dynamic.163data.com.cn Los Angeles, California QuadraNet Networks 83.33% 384.906ms 384.906ms 384.906m
2 months ago, Data centres in LA were considered as one of the most friendly place where you buy VPSs from if you use them for US-China business. The latency was always around 180-200ms. Now as you can see, in stead of going straight to West Coast, the worst routing has been chosen for LA. I don't think this is the US end problem. I think it's got something to do with China telecom. It's their routing preference problem. In other words, they do this intentionally.
@ kyaky I just used 17ce.com and traced to our Los Angeles location (MultaCOM).
http://17ce.com/site/http/201310_91f7b8a59d0a38dfddbbc7941ff270fb.html is the report. All China Telecom is going via Savvis (Century Link) and is doing fine.
Note that we don't use Quadranet.
This isn't an LA problem.
I know, I'm not talking about the service you provide. I'm just saying there's gotta be something wrong with China telecom side. It seems like China telecom is making routing trouble to the known IP ranges from West Coast especially for LA IPs.
Don't let them find out your IP range otherwise your customers are going to Ireland too... xD
There's a peering dispute (likely asking for more money) between China Telecom and whoever it is - possibly Tinet. Nothing wrong with China Telecom itself. A few other carriers go fine to China Telecom.
EDIT: http://onesc.net/communities/as3257/
65000:XXXX do not announce to ASXXXX
Set XXXX to China Telecom. Do not announce Tinet to China Telecom until fixed. The end.
I quite agree. It doesn't affect me actually. I'm a Chinese living in Australia. but when I see that many Chinese friends panic about this routing problem in China when people wouldn't even want to try to complain because they know the China telecom and related government department would not bloody care about what they complained. that makes me sad. what can they do.
Now it seems people from China prefer to have VPS from Seattle, Las Vegas and San Francisco, those cities except LA that are close to West Coast because these IPs provide ping around <200ms without worrying about routing to Ireland.
@concerto49 if you believe it's got nothing to do with China telecom end, you can use http://ping.chinaz.com/ to test your ip 192.3.172.173 . You will see what I mean. You can also see from traceroute, while the routing is still within China, the ping jumps from 2ms to 250ms (two close gateways next to the backbone router, updated filter system? I don't know) which is ridiculous. If you think China doesn't manipulate routing, you probably have to dig more information about what's going on in China atm. If customers from china complain about the networks, I think you just ask them to send a ticket to China telecom. No other cure.
I'm only referring to the routing to Europe madness. China internally always has congestion issues. That's widely known. You can never have enough bandwidth in China.
I think there are plenty of reasons why public protest of a telecom might be a bit muted in China
Chinese market is also much more tolerant of these types of issues. As concerto49 already pointed out, there are always domestic network congestion issues in China.
Fiber cuts or not - China routing always sux
I think they do it on purpose, so the Chinese people get less of the bad western influence.
I already put on my Tinfoil hat, what more do you want us to do?
CT itself is 'fine,' in the sense that they want more cash and not give away free peering. Now, large scale telcos like Tinet / Level3 are obviously not very interested in this, since the very thing that makes them tier 1 is their settlement free peering.
So, they do the only things they can do -- either route it to an IXP where they have another session with CT for peering, be that in Europe, or dump it to an upstream of CT (Hot potato routing, essentially) and make it not their problem
Regardless, unicom has 2/3 fiber paths cut, and is now rerouting everything. This is, sadly, dumb (Or great if your grief is with Chinese DDoS).
as @kyaky mentioned, yes, in recent 2 months the china telecom to some LA IPs route via Ireland then US west, so i cancelled/not to purchase many VPS from LA providers. still keep some from Las Vegas, San Diego, San Jose and Seattle, don't know if this chine unicom issue would cause some worse condition...
A lot of traffic from San Diego, San Jose and Seattle come from Los Angeles. One set of cables land near Los Angeles and that's where it all runs off. There would be no difference.
The difference is in the transit carriers used in the network mix of a VPS provider. That's all.
There is only one US-China submarine fiber. Most asian carrier use that fiber. And if that fiber is cut(which happens pretty frequently so force them to build a second fiber) everybody is affected.
@kyaky
Tinet (among others) is routing CT traffic through EU (Via their Ireland core) due to issues in LA.
But yeah.
I think if it is a dispute or issue inside China, routing from China to anywhere in US should be affected. I just test my machines near Chicago and routing seems normal. So could be a LA local issue.
Looks like China Telecom just rerouted traffic to nlayer. I had terrible packet loss for 3 days(Tinet). It is a lot faster now with nlayer and no packet loss.
traceroute to x.com (198.46.x.x), 30 hops max, 60 byte packets
1 192.168.2.1 (192.168.2.1) 0.238 ms 0.243 ms 0.224 ms
2 61.140.196.1 (61.140.196.1) 2.474 ms 2.517 ms 2.936 ms
3 113.98.94.73 (113.98.94.73) 5.910 ms 5.912 ms 6.021 ms
4 121.8.90.25 (121.8.90.25) 3.554 ms 3.551 ms 3.659 ms
5 58.61.216.45 (58.61.216.45) 7.502 ms 7.582 ms 7.671 ms
6 202.97.35.254 (202.97.35.254) 4.408 ms 6.813 ms 6.838 ms
7 202.97.60.66 (202.97.60.66) 6.825 ms 13.445 ms 14.914 ms
8 202.97.58.222 (202.97.58.222) 276.054 ms 276.087 ms 276.229 ms
9 202.97.90.10 (202.97.90.10) 162.675 ms 162.720 ms 162.711 ms
10 xe-6-0-0.ar1.lax2.us.nlayer.net (69.31.127.21) 205.356 ms 205.371 ms 205.359 ms
11 as29761.xe-1-0-2.ar1.lax2.us.nlayer.net (69.31.121.254) 180.555 ms as29761.xe-5-0-1.ar1.lax2.us.nlayer.net (69.31.127.42) 286.983 ms as29761.xe-1-0-2.ar1.lax2.us.nlayer.net (69.31.121.254) 179.338 ms
12 colo-lax8 (96.44.180.50) 172.788 ms colo-lax8 (96.44.180.42) 224.393 ms 173.170 ms
13 67.215.251.214.static.quadranet.com (67.215.251.214) 174.698 ms 175.528 ms 175.179 ms
14 host.colocrossing.com (192.210.x.x) 181.153 ms 182.299 ms 181.646 ms
15 host.colocrossing.com (192.210.x.x) 173.869 ms 173.921 ms 173.718 ms
16 host.colocrossing.com (198.46.x.x) 170.837 ms 171.159 ms 171.204 ms
170ms? That's very fast
if you take a look at here" 8 202.97.58.222 (202.97.58.222) 276.054 ms 276.087 ms 276.229 ms", it's not. the internal is fvcked
We don't know the CU cuts are true or not. There is no official news, or traceroute to prove it yet.
The CT->Tinet issue has been there for months.
I talked to GTT NOC about why CT routing traffic to their EU path back in August, they said(direct quote)
"We peer with China Telecom in multiple places around the world and that has not changed. Any type of route changes like that are on China Telecom side and not influenced by Tinet at all (they choose where to pass traffic to Tinet). If you need any more information please let us know."
Then I did couple tests, I announced a /24 to Tinet, and run MTR from a CT server to the IPs on that /24, I saw the traffic goes to Tinet SJC/LAX, but there are lots packets loss and high latency especially to LAX. then after some minutes, all the sudden the path changed, routes CT traffic to Tinet EU. I stopped announcing that /24 for couple minutes and start announcing it again, and the traffic goes routes to SJC/LAX again, then sometime later, goes to EU.
So the explanation is simple, CT is known to do dynamic routing, they would try to route over congested paths automatically.
The peering capacity between Tinet US and CT is saturated badly, that's why they changed the path and route traffic to Tinet EU. I guess that path is also saturated. Tinet needs to increase peering with CT to solve this issue, otherwise it's gonna get worse.
After GTT acquisition of Tinet, I'm seeing things at Tinet have been going down hill fast, constant backbone issues, peering issues, etc. They should really get the quality up while selling transit cheap.
For providers, just use the BGP community to have Tinet stop announcing to CT.
And routing to Ireland provides a better influence?!?! :-)
@jimpop no, but making the internet as slow as possible decreases the chances of influence
Looks like Quadranet is now routing some stuff via Nlayer, certainly latency much better now.
Internet Initiative Japan (IIJ, cool-looking acronym right) does the same. From Ablenet I normally have a pretty good latency with a "Japan -> London" jump on the traceroute:
I suppose this is a cable via middle-east or Russia (less likely).
And it looks like this cable does not have enough capacity, so during the peak load periods (mostly weekends) they switch to a different path, one which goes from Japan all the way via the US (to LAX first), and then the latency increases quite a bit.
It is just a slow path and delay did not carry forward.
Previously, routing from China to US by CT heavily relied on nLayer while they use Tinet too but not very often. I guess there are some changes after the merger, so that a backup route (Tinet) suddenly becomes a major route and the capability that was designed for backup cannot handle the majorities routes.
If there were any changes, they would be on GTT(Tinet) side. I've noticed a lot of things changed(towards the bad side) after Tinet -> GTT. Minor backbone issues at least every other day, Major issues every month, some of them even last hours if not days. There were a backbone issue around Phoenix,AZ yesterday lasted atleast 10 hours, and they blamed on Level3's fiber. Their SLA is just a joke.
rm_, I have a feeling that's most likely a trans-Russian path from Japan to Europe due to the latency. From what I've read online Europe to Singapore is ~<200ms and Europe to Japan via Russia is ~<200ms (from London). Any trans-US path to London from Tokyo is at least 260ms. (100ms Tokyo to LA, LA to London is 160ms). It's not going to be the Middle East submarine path (via Singapore) because that adds Tokyo to Singapore of about ~70ms.
You can view their network coverage here:
http://eng.hk.chinaunicom.com/networkcoverage_d.html
http://eng.hk.chinaunicom.com/networkcoverage.html
I also know that for TiNet they use European (Italian and German) IP addresses in some of their core routers in the US likewise Level3 uses some American IP addresses for their core routers in Europe. In these multinational companies you sometimes can't trust the IP addresses country of origin in the WHOIS database to determine where it is, its most likely a guess. Likewise you can use the rDNS but that can be mis-labeled when their doing upgrades/shifting capacity (ala Comcast, Level3)...
It would seem a logical theory, and one I once thought was true/common. But that's not whats happening here, nor "the interval is fcked". Backbone providers will often group router IPs into a handful of subnets, and even though that particular router sits on the path when it returns the ICMP packet it is taking a different route than a return packet from the destination due to the what the peer will accept, either that or it is extremely slow at processing ICMP packets. However the final destination showing a ping time of ~170ms would be accurate, unless you have a seriously out clock. There is an excellent paper that helps draw these points out by the nlayer CTO.