New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@kris
Packet loss galore though!
Also no ipv6 routing.
Main site/billing seems to be back up
Also some servers still have no routing, I have one with routing, the other is still no go.
I'm still unable to mtr (still stopping at Telia in ATL) but zabbix is picking up data again and so are other various sources.
Their IPs are back online for us so we can get to some of our servers, of course our IPs are not being announced so no chance of me going to sleep early tonight.
Any RFOs yet?
hmm IP back online their website also back online
Spoke too soon, took 1 monitor as gospel. 2/3 down.
Routes and prefixes are not routing, including bog standard monthly rentals.
Maybe they need to wake up level 3 network tech, or rebuilding router config? Something's gone to hell.
Outside:
http://uptime.munroenet.com/index.php
Things aren't even routing properly internally:
https://munroenet.com/uptime/
As you can see, things are still clearly broken.
Quite.
Taking Charter out, that's a first.
What I can guess: their major uplinks are down / bit the dust.
It's all running on one congested Charter uplink (maybe a 1Gbps after-thought for OOB usually) that does not have ACL setup for their main subnets.
Ergo, packet loss and many prefixes down. Does not answer why Level3, etc. pissed out. F
or $25/month I'll watch MTR as things get fixed while STFU and letting them do the needful.
Tends to happen when your sole Charter backhaul from ATL -> Lenoir goes down...
Everything will come back up.. as soon as Charter does
My server apparently just came back up.
And we're back online.
Everything is back online now.
Back online now too 2hours 39mins downtime.
Everything online, total full outage was 49 mins for me but had packet loss until 3am EST.
UPDATE: Our Atlanta based upstream provider has advised us that they are conducting a transport upgrade maintenance that was previously thought not to include the CentriLogic circuit. CentriLogic was not notified of the maintenance window, which was scheduled to last until 06:00 AM EST. We have escalated with the upstream to restore service as soon as practical, but the maintenance is currently underway. They are aware of the unintended impact and will expedite the work as much as possible to restore service.
According to uptime robot, my 1 vm with them was:
Down:5:24 am
Up:7:59 am
We did have some network issues last night. I'll post all of our updates from Facebook last night.
Update #1: Multiple upstream providers originating in Charlotte, NC have been affected by a region-wide fiber cut. Our alternate provider originating in Atlanta is also experiencing an issue at present. We are currently using a limited capacity local circuit which has restored some connectivity to the Data Center. We are working with the Atlanta provider to restore connectivity.
UPDATE 2: Our Atlanta based upstream provider has advised us that they are conducting a transport upgrade maintenance that was previously thought not to include the CentriLogic circuit. CentriLogic was not notified of the maintenance window, which was scheduled to last until 06:00 AM EST. We have escalated with the upstream to restore service as soon as practical, but the maintenance is currently underway. They are aware of the unintended impact and will expedite the work as much as possible to restore service.
After that it looks like the backup link was restored at around 4:00 AM EST. The regional fiber cut was to be repaired by 9:00 AM EST as we were informed.
Customers may request a PIR via a ticket through Hostbill and we will provide it once it is completed. Anytime you're unable to reach the site you can always submit a ticket via email to [email protected] via your registered email.
Any plans to add some type of redundancy to your unprotected network? Fiber cut & maintenance in one night, at around the same time... tough break.
Any reason all your maintenance windows & downtime coincide with Charter maintenance? Could be worth investing in some additional non-Charter backhaul.
I wonder if you read the UPDATE 2? Like how are you expected to know your backup upstream is doing a maintenance when they didn't tell you and didn't even include you into their plans. I say the Atlanta upstream fucked up here. No fault here for @dacentec.
What is a PIR?
What's a PIR?
Pain in (the) Rear.
Oh I read it, I just don't buy it.
What if there is no real backup? There was an outage this summer which Dacentec blamed on Charter dropping a 100Gig bundle between Charlotte & Atlanta - which appears to be exactly what happened last night.