New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
All of our stuff is inaccesible since 12:54pm (it's now 1:09pm). I don't know why yet.
Even our website/billing/vpscp server, which is separate hardware, separate network hardware, separate uplink, different rack, different power feed, but same datacenter, is down/inaccessible.
Yes, our bills are paid.
Tracerouting any of our stuff seems to drop packets outside of our network. Not sure if there's some kind of weird networking thing going on.
Sent Chris a txt, redirected to open a ticket with Luc, who should be on it.
If anyone can tell ME that would be great too.
Well, I noticed the billingcp/vpscp was also down. Thanks for the quick response, eagerly awaiting updates. Good luck!
Is that stuff in different datacenters?
It's not.
Our NY stuff is up, so I doubt it's network related.
Internally?
@Damian Here's what i get when i try to traceroute citrine:
traceroute to citrine.ipxcore.com (198.144.190.5), 30 hops max, 38 byte packets
1 89.205.76.1 (89.205.76.1) 10.183 ms 11.958 ms 9.378 ms
2 89.205.126.1 (89.205.126.1) 10.688 ms 10.777 ms 16.913 ms
3 195.26.158.209 (195.26.158.209) 12.269 ms 13.457 ms 16.752 ms
4 84.1.246.89 (84.1.246.89) 27.322 ms 13.616 ms 15.298 ms
5 238.131.158.95-rev.novatelbg.net (95.158.131.238) 17.391 ms 17.776 ms 14.522 ms
6 te4-8.ccr01.sof02.atlas.cogentco.com (149.6.68.73) 233.761 ms 214.019 ms 204.435 ms
7 te2-6.ccr01.beg01.atlas.cogentco.com (154.54.73.89) 59.590 ms te3-1.ccr01.beg01.atlas.cogentco.com (154.54.73.93) 54.047 ms te3-2.ccr01.beg01.atlas.cogentco.com (154.54.73.97) 61.663 ms
8 te0-1-0-3.ccr21.bud01.atlas.cogentco.com (154.54.63.197) 67.165 ms te0-3-0-3.ccr21.bud01.atlas.cogentco.com (154.54.63.201) 52.961 ms te0-2-0-3.ccr21.bud01.atlas.cogentco.com (154.54.63.210) 51.053 ms
9 te0-0-0-3.ccr21.bts01.atlas.cogentco.com (130.117.3.53) 57.875 ms te0-4-0-4.ccr22.bts01.atlas.cogentco.com (154.54.57.253) 52.291 ms te0-4-0-4.ccr21.bts01.atlas.cogentco.com (154.54.57.249) 54.879 ms
10 te0-4-0-5.ccr22.muc01.atlas.cogentco.com (154.54.74.182) 51.506 ms te0-0-0-2.ccr21.muc01.atlas.cogentco.com (130.117.51.182) 51.585 ms te0-3-0-5.ccr22.muc01.atlas.cogentco.com (154.54.39.9) 63.958 ms
11 te0-3-0-5.mpd21.fra03.atlas.cogentco.com (130.117.50.233) 50.420 ms te0-2-0-2.ccr22.fra03.atlas.cogentco.com (130.117.50.245) 57.895 ms te0-3-0-2.mpd21.fra03.atlas.cogentco.com (130.117.2.205) 70.134 ms
12 te0-5-0-2.mpd22.ams03.atlas.cogentco.com (154.54.39.189) 63.402 ms te0-2-0-2.ccr21.ams03.atlas.cogentco.com (130.117.1.162) 60.536 ms te0-0-0-2.ccr22.ams03.atlas.cogentco.com (130.117.3.89) 59.366 ms
13 te0-0-0-7.ccr21.lpl01.atlas.cogentco.com (154.54.37.74) 70.713 ms te0-2-0-2.ccr22.lpl01.atlas.cogentco.com (154.54.37.118) 67.296 ms te0-2-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.37.86) 64.161 ms
14 te0-5-0-4.ccr22.ymq02.atlas.cogentco.com (154.54.87.58) 139.190 ms te0-2-0-4.ccr21.ymq02.atlas.cogentco.com (154.54.0.70) 151.209 ms te0-4-0-4.ccr22.ymq02.atlas.cogentco.com (154.54.44.213) 143.860 ms
15 te0-3-0-5.mpd22.yyz02.atlas.cogentco.com (154.54.43.217) 152.380 ms te0-1-0-5.ccr21.yyz02.atlas.cogentco.com (154.54.44.98) 144.639 ms te0-2-0-7.mpd22.yyz02.atlas.cogentco.com (154.54.42.225) 146.606 ms
16 te8-1.ccr01.buf02.atlas.cogentco.com (154.54.27.70) 288.966 ms te4-3.ccr01.buf02.atlas.cogentco.com (154.54.27.66) 262.696 ms te7-1.ccr01.buf02.atlas.cogentco.com (154.54.43.66) 195.430 ms
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29
traceroute to citrine.ipxcore.com (198.144.190.5), 18 hops max, 40 byte packets
1 88.85.125.49 1.222 ms 1.042 ms 1.026 ms
2 80.77.149.190 1.203 ms 1.337 ms 1.024 ms
3 80.77.144.165 0.574 ms 0.403 ms 0.565 ms
4 212.200.235.113 6.256 ms 6.339 ms 6.172 ms
5 212.200.6.162 6.489 ms 6.498 ms 6.491 ms
6 212.200.6.186 6.338 ms 6.344 ms 6.340 ms
7 213.248.82.157 14.454 ms 14.297 ms 14.289 ms
8 80.91.250.68 21.788 ms 21.784 ms 21.782 ms
9 213.155.132.240 40.502 ms 40.505 ms 40.506 ms
10 213.155.131.147 166.872 ms
213.155.131.145 224.302 ms 135.363 ms
11 80.91.246.36 145.508 ms 144.653 ms 145.657 ms
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
For me it dies at
buf-b1-link.telia.net (80.91.246.36) 126.728 ms 126.476 ms 126.672 mswhen tracerouting citrine.ipxcore.com. I guess that means the traffic doesn't even reach the DC/exit the upstream network?That's what we're seeing too. Our switch has an IP address, and tracerouting to it dies too.
Issue resolved? My IRC bot is up again at least
don't think so, -my server- as well as ipxcore.com are still down.
Server is up
Looks like some kind of weird partial power outage? Some servers have uptime of ~4 minutes, some servers have uptime of ~50+ days?
More info soon
@Damian citrine has gained back connectivity
My VPS node uptime says 11:54:03 up 19 days, 11:20, 2 users, load average: 0.00, 0.00, 0.00 (since i rebooted it the last time as far as i recall)
traceroute to citrine.ipxcore.com (198.144.190.5), 30 hops max, 38 byte packets
1 89.205.76.1 (89.205.76.1) 17.268 ms 92.581 ms 15.242 ms
2 89.205.126.1 (89.205.126.1) 74.918 ms 67.662 ms 16.637 ms
3 195.26.158.209 (195.26.158.209) 67.930 ms 48.924 ms 15.038 ms
4 84.1.246.89 (84.1.246.89) 25.806 ms 32.005 ms 15.605 ms
5 238.131.158.95-rev.novatelbg.net (95.158.131.238) 18.490 ms 17.632 ms 147.251 ms
6 te4-8.ccr01.sof02.atlas.cogentco.com (149.6.68.73) 84.453 ms 194.759 ms 255.433 ms
7 te1-6.ccr01.beg01.atlas.cogentco.com (154.54.73.85) 77.507 ms te3-2.ccr01.beg01.atlas.cogentco.com (154.54.73.97) 76.565 ms te3-1.ccr01.beg01.atlas.cogentco.com (154.54.73.93) 94.997 ms
8 te0-3-0-3.ccr21.bud01.atlas.cogentco.com (154.54.63.201) 73.559 ms te0-1-0-3.ccr21.bud01.atlas.cogentco.com (154.54.63.197) 81.296 ms te0-4-0-3.ccr21.bud01.atlas.cogentco.com (154.54.73.81) 52.971 ms
9 te0-2-0-3.ccr21.bts01.atlas.cogentco.com (154.54.36.10) 143.154 ms te0-0-0-3.ccr22.bts01.atlas.cogentco.com (130.117.49.146) 64.025 ms te0-1-0-3.ccr21.bts01.atlas.cogentco.com (130.117.3.57) 73.211 ms
10 te0-0-0-2.ccr22.muc01.atlas.cogentco.com (130.117.0.42) 75.520 ms te0-1-0-2.ccr21.muc01.atlas.cogentco.com (130.117.50.122) 83.879 ms te0-2-0-2.ccr22.muc01.atlas.cogentco.com (154.54.38.253) 59.436 ms
11 te0-5-0-4.ccr21.fra03.atlas.cogentco.com (154.54.74.206) 70.868 ms te0-2-0-2.mpd22.fra03.atlas.cogentco.com (130.117.50.241) 108.338 ms te0-3-0-5.mpd21.fra03.atlas.cogentco.com (130.117.50.233) 53.396 ms
12 te0-0-0-2.mpd22.ams03.atlas.cogentco.com (130.117.0.133) 75.445 ms te0-2-0-2.mpd21.ams03.atlas.cogentco.com (130.117.2.202) 79.501 ms te0-3-0-2.ccr22.ams03.atlas.cogentco.com (154.54.39.193) 67.895 ms
13 te0-3-0-7.ccr21.lpl01.atlas.cogentco.com (154.54.37.98) 80.769 ms te0-2-0-2.ccr22.lpl01.atlas.cogentco.com (154.54.37.118) 99.852 ms te0-6-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.38.50) 70.034 ms
14 te0-3-0-4.ccr22.ymq02.atlas.cogentco.com (154.54.0.74) 147.180 ms te0-4-0-3.ccr21.ymq02.atlas.cogentco.com (154.54.44.205) 139.368 ms te0-1-0-4.ccr22.ymq02.atlas.cogentco.com (66.28.4.21) 144.824 ms
15 te0-1-0-6.mpd22.yyz02.atlas.cogentco.com (154.54.42.233) 146.049 ms te0-6-0-7.ccr21.yyz02.atlas.cogentco.com (154.54.25.29) 223.102 ms te0-6-0-1.ccr21.yyz02.atlas.cogentco.com (154.54.44.114) 160.530 ms
16 te7-1.ccr01.buf02.atlas.cogentco.com (154.54.43.66) 162.432 ms te3-1.ccr01.buf02.atlas.cogentco.com (154.54.43.62) 145.538 ms te8-1.ccr01.buf02.atlas.cogentco.com (154.54.27.70) 221.082 ms
17 38.122.36.46 (38.122.36.46) 189.908 ms 149.942 ms 294.521 ms
18 ipxcore.aggr.buffalo.nwnx.net (75.127.1.82) 229.537 ms 149.620 ms 283.533 ms
19 citrine.ipxcore.com (198.144.190.5) 275.581 ms 223.537 ms 287.152 ms
traceroute to citrine.ipxcore.com (198.144.190.5), 18 hops max, 40 byte packets
1 88.85.125.49 1.122 ms 1.821 ms 1.033 ms
2 80.77.149.190 1.198 ms 1.032 ms 1.074 ms
3 80.77.144.165 0.581 ms 0.426 ms 0.401 ms
4 212.200.235.113 6.192 ms 6.238 ms 6.179 ms
5 212.200.6.162 6.499 ms 53.458 ms 35.744 ms
6 212.200.6.186 6.342 ms 6.384 ms 6.346 ms
7 213.248.82.157 14.466 ms 14.310 ms 14.303 ms
8 80.91.250.68 21.933 ms 44.262 ms 21.940 ms
9 213.155.132.240 118.662 ms 40.509 ms 76.372 ms
10 213.155.135.61 136.139 ms 136.109 ms 138.645 ms
11 80.91.246.36 144.569 ms 144.547 ms 144.577 ms
12 213.248.96.42 156.446 ms 164.853 ms 161.105 ms
13 75.127.1.82 136.336 ms 137.399 ms 136.208 ms
14 198.144.190.5 155.720 ms 156.585 ms 157.051 ms
It is possible, if the switch/router, was on the powered down area in case of a partial problem, everything went offline, tho not all servers lost power.
Node citrine stayed powered on.
Nodes diamond, sapphire, opal, emerald were hard-rebooted without proper shutdown.
Sapphire, opal, emerald are up and running.
Diamond has filesystem corruption, /vz remounted readonly pending IP KVM access.
offtopic @Janevski Macedonia? /offtopic
Macedonia.
offtopic @janevski Hello, Neighbor ;-) /offtopic
What DC is this @Damian?
diamond - 12:00:52 up 23 min, 2 users, load average: 3.23, 3.13, 2.25
citrine - 15:00:54 up 49 days, 23:16, 1 user, load average: 7.77, 5.90, 5.23
opal - 15:00:56 up 23 min, 3 users, load average: 12.72, 8.51, 4.59
emerald - 15:00:57 up 23 min, 1 user, load average: 3.16, 3.42, 4.67
sapphire - 12:00:59 up 23 min, 1 user, load average: 3.37, 1.92, 1.74
cpanel2 - 12:00:59 up 46 days, 16:01, 0 users, load average: 1.04, 1.12, 1.26
jasper - 15:01:02 up 23 min, 0 users, load average: 11.63, 6.22, 3.79
We're still missing carbon.
colocrossing buffalo
Oh gawd i hope they didn't accidently hit your power when they were racking my new server... sorry!
If not, that would explain why I'm still waiting.
Hello there!
[We'll talk via PM.]
Successfully umounted filesystem, fsck running now.
fsck finished, booting containers on this node now.
Thank you once again for the frequent status reports and quick responses.
ColoCrossing FTW - whats the deal with unannounced power outages? Is there a RFO?
We haven't really had an issue like this in the 9 months we've been with them. So yes, FTW.
I'm sure there is. I'll request it once we get everything tied up.
I didn't have an outage.
Same, mostly.
We had a route blip it seems, but it lasted less than a minute (the smallest time unit nodeping runs). I'm assuming the switch we're uplinked to was rebooted due to whatever power blip happened and came back very quickly.
I'm wondering if that same UPS is giving Jon a hard time again. I know they had a UPS that caused some headaches a few months ago.
Francisco