All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Routing around the hurricane?
Anyone noticing routing weird routes in / around areas that go through New York?
Look at this from Buffalo:
Buffalo to Cleveland to Chicago to Washington, DC to Chicago to New York
3: buf-b1-link.telia.net 0.682ms asymm 5
4: cle-b1-link.telia.net 6.270ms asymm 6
5: chi-bb1-link.telia.net 13.348ms asymm 7
6: xe-9-2-0.edge1.washington4.level3.net 13.155ms
7: ae-31-51.ebr1.Chicago1.Level3.net 19.985ms asymm 5
8: ae-2-2.ebr2.NewYork2.Level3.net 33.383ms asymm 7
9: ae-6-6.ebr2.NewYork1.Level3.net 21.365ms asymm 6
10: ae-62-62.csw1.NewYork1.Level3.net 22.924ms asymm 7
11: ae-4-90.edge4.NewYork1.Level3.net 21.359ms asymm 9
12: NEW-YORK-UN.edge4.NewYork1.Level3.net 21.970ms asymm 9
Take it Telia has equipment/facility down in Manhattan? This isn't the normal route we see to NYC.
Comments
Routing to our node in NY (Telx, DC #1)
Equinix Chicago
core1.chi1.he.net> traceroute 199.38.181.XXX
Tracing the route to IP node (199.38.181.XXX) from 1 to 30 hops
1 11 ms 3 ms 1 ms 69.31.111.109
2 <1 ms <1 ms <1 ms 69.31.111.137
3 23 ms 24 ms 24 ms 69.22.142.33
4 43 ms 33 ms 17 ms 69.31.31.154
5 31 ms 24 ms 25 ms 69.22.142.38
6 25 ms 24 ms 24 ms 69.31.34.226
7 25 ms 25 ms 25 ms 199.38.181.XXX# Entry cached for another 58 seconds.
Equinix Dallas
core1.dal1.he.net> traceroute 199.38.181.XXX
Tracing the route to IP node (199.38.181.XXX) from 1 to 30 hops
1 * * * ?
2 127 ms 124 ms 125 ms 69.31.63.125
3 50 ms 49 ms 49 ms 69.22.142.113
4 20 ms 24 ms 25 ms 69.22.142.118
5 34 ms 41 ms 33 ms 69.22.142.106
6 39 ms 49 ms 56 ms 69.22.142.142
7 44 ms 48 ms 49 ms 69.31.95.174
8 50 ms 49 ms 49 ms 69.31.34.226
9 49 ms 49 ms 50 ms 199.38.181.XXX# Entry cached for another 51 seconds.
Equinix Los Angeles
core1.lax1.he.net> traceroute 199.38.181.XXX
Tracing the route to IP node (199.38.181.XXX) from 1 to 30 hops
1 26 ms 16 ms 19 ms 206.223.123.61
2 72 ms 74 ms 75 ms 69.22.142.115
3 71 ms 74 ms 75 ms 69.22.142.38
4 76 ms 73 ms 75 ms 69.31.34.222
5 80 ms 74 ms 75 ms 199.38.181.XXX# Entry cached for another 59 seconds.
Given how many DC's in NYC are down or having major issues i'm not surprised
Francisco
I saw some packet loss on he.net between USA and Europe, but this might be normal for them.
only the bad ones...
WebNX
3 24 ms 24 ms 23 ms 10.245.203.225
4 34 ms 30 ms 42 ms te0-6-0-2.er11.thlon.ov.easynet.net [89.200.135.31]
5 28 ms 27 ms 36 ms 10gigabitethernet1-1.core1.lon1.he.net [195.66.224.21]
6 43 ms 35 ms 40 ms 10gigabitethernet2-4.core1.par2.he.net [72.52.92.42]
7 119 ms 118 ms 128 ms 10gigabitethernet7-1.core1.ash1.he.net [184.105.213.93]
8 134 ms 145 ms * 10gigabitethernet5-4.core1.chi1.he.net [184.105.213.245]
9 164 ms 162 ms 171 ms 10gigabitethernet3-2.core1.den1.he.net [184.105.213.86]
10 * 196 ms 189 ms 10gigabitethernet1-4.core1.lax2.he.net [72.52.92.38]
11 190 ms 190 ms 188 ms 216.218.213.250
12 190 ms 190 ms * 100-42-223-150.static.webnx.com [100.42.223.150]
13 191 ms 189 ms 188 ms 100-42-223-75.static.webnx.com [100.42.223.75]
14 190 ms * 189 ms 142-4-49-108.static.webnx.com [142.4.49.108]
Trace complete.
The DC that our master is in just happens to be down. #SandyYouSuck
We're currently getting pings that are about 100ms longer than normal in Europe. We're colocating in San Jose, California, but I assume that the longer pings are because of Hurricane Sandy, right?
Can I blame the hurricane for 30mbit speed from Denver to Dallas? Because I'd like to.
Appears to be back to normal from Buffalo to NYC. Wasn't around here this afternoon to keep an eye on things:
3: buf-b1-link.telia.net 3.296ms asymm 5
4: nyk-bb1-link.telia.net 9.961ms asymm 6
5: nyk-b5-link.telia.net 10.052ms asymm 6
6: level3-ic-120265-nyk-b5.c.telia.net 10.451ms asymm 9
7: ae-3-80.edge4.NewYork1.Level3.net 12.438ms asymm 9
8: NEW-YORK-UN.edge4.NewYork1.Level3.net 14.354ms asymm 9
9: NYUGWA-VL10.NET.NYU.EDU 15.836ms asymm 12
10: WSQDCGWA-VL901.NET.NYU.EDU 22.483ms asymm 13
Appears Telia was probably fine but that Level3's handoff in NYC was down earlier.
The majority of the DCs that are down had redundancy...
Power went of course went out and they had to go to generator power. Problem is that the fuel storage was flooded out and fuel could not be pumped up to the generators. This resulted in about 6 hours of fuel for the generators to run, before they ran out and had to shutdown the DC totally. Either that or their generators were flooded out, because they located in the basement...
So, they do have redundancy. Although pretty stupid how they put everything for it in the basement or ground level, so the equipement won't be useful in a high water situation.
That is State Code
the truth. everyone and their grandmother's generators were flooded out
State codes are pretty moronic (about generators being subterranean) and need changed in NY after this.
Glad I didn't colo gear in NYC. Always deemed it a very big risky place.
Right now NYC is having a once every decade issue that impacts infrastructure.
What was the last one? 9/11?
Yeppers @Nick
Bound to see some issues in NYC infrastructure from this flooding far into the future.
Media already bemoaning the outdated infrastructure and lack of maintaining Manhattan.
Updates from our DC in NY (Telx NYC1)
So as you can see not all DCs went down
I'd actually like to see the amount of providers that have a secondary master setup, prior to me posting this... not many.