New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
They sent out a maintenance notice earlier if I recall.
Crazy if they acrually planned for their main website to go down with the servers
really? can you tell me when the notification was given?
I will try to check my email again.
Is there any information on how long the maintenance will take?
I checked my email again. I found only a notification on September 18th about their client portal maintenance.
There is no other related maintenance.
From what I can see, they've stopped announcing a lot of IP's. Willing to bet someone made a bad network update.
https://status.centrilogic.com/incidents/c96vlybkl12q
Thanks for the very useful link.
How bad is the situation there? Maybe you have some information to share?
Sad, you guys will never believe that, i got my production website down in the middle of a demo for interested buyer ... hope dacentec get back up soon
How widespread was the impact of this event?
The region got hit hard from the rain and wind for the hurricane. Flooding has been bad in my area and I'm like 90 mins from Lenoir. Trees have been coming down like crazy too. Several hospitals are being evacuated do to flash floods.
I found the problem. They are relying on diesel generators. People always say new technology is better. The hamsters never let me down before, recommend picking a data center that is damaged powered. Also better for the environment.
But fun aside. Dacentec is really solid. Reach out to them post issue for comp and all. They are always professional and very reasonable. I am no longer a customer but I had the BEST experience with them. Reminded me a little of back in the day when we had ThePlanet. RIP
How long do you think it will take to fix it? Have you heard of similar treatment elsewhere?
We've been down 12 hours now. I wonder if they had zero backup power?!
"1 customer without power" at their address. https://imgur.com/a/ELBvOfo
I read they have backup power.
But the problem is in the network. Maybe the network around the datacenter is cut off.
But the problem is we don't know how much progress has been made.
I hope they can provide more detailed information so we can decide what action needs to be taken.
There's a lot of stuff under water in western NC due to record flooding. I believe they backhaul their traffic to Atlanta, likely with the fiber running right through some of the places hardest hit with the storm. Our HQ is in the Charlotte area, which fortunately was mostly spared from the worst impacts of the storm. But, since we're down river from many of the areas that were flooded, we are seeing some flooding here.
If anyone needs a temp server, DM me. QuickPacket has gear available in all of our locations, and we'll waive costs for a week to get you back on your feet.
Our policy is to start disaster recovery procedure after a predetermined period of downtime.
The period is predetermined for each application and it's unrelated to the cause of downtime.
Unforeseen natural disaster. In Vietnam, we have also experienced flooding, power outage and internet loss for more than 2 weeks. Hope you are getting through this.
Below is from their Facebook:
Update 9/27/24 - 17:17
We are actively collaborating with our partners to resolve the ongoing situation as swiftly as possible.
Our carrier partners are in communication with the power companies, performing repairs and will provide updates at the earliest opportunity.
We are closely monitoring the situation and will continue to share periodic updates. We sincerely appreciate your patience and understanding during this challenging time.
Update:
We want to ensure everyone that our data center has remained operational and has had power, although it has been intermittent. Our backup diesel generators have been running as needed. The high-efficiency UPS systems are also functioning as expected to provide stability during any power fluctuations.
However, the external infrastructure surrounding the data center has been significantly impacted by the hurricane. This includes widespread power outages and severe damage to carrier networks, which has affected connectivity. Despite our multiple entry points, the external carriers’ infrastructure has not been fully restored.
We are working closely with our partners to resolve this as quickly as possible, but their repair efforts are being delayed due to the extensive damage in the area. We are monitoring the situation closely and will continue to provide updates periodically.
We greatly appreciate your patience and understanding during this challenging time.
——End of Update——
Dear Valued Customer,
We regret to inform you that due to a severe storm impacting the region, our Lenoir data center has experienced a loss of connectivity. Our team is working closely with our partners and our carrier providers to restore service as quickly as possible.
We understand the inconvenience this may cause, and we deeply appreciate your patience and understanding during this time. Please be assured that we are doing everything in our power to minimize the disruption, and we will do our best to continue to provide updates as the situation develops.
Sincerely,
Customer Support Team
Dacentec
https://www.facebook.com/Dacentec/posts/pfbid02SBTmzsMwsnmy6VpXyMB5WnjcpMn7Yg8whwGXDD2ukH4th4DUV3kUSF7ZtAQ2QShnl
The question is how long can this downtime last?
Is it less than 24 hours? Or more?
The information provided does not provide detailed information on how far the repair has been done.
This will help the user decide what to do next.
If it was a fiber cut, which at this point I would think has to be the case since it's been down this long, depending how many fibers were cut and the conditions around where it was cut, it can be hours to days to repair (as someone has to manually, one-by-one repair the fibers), unfortunately. I am sure they are freaking out as much as we are and are working on some alternatives, just in case, however, the issue they probably face at this time is capacity, not whether they can get 'some connectivity'.
my 2 cents.
Cheers!
Are there any data centers around Florida and North Carolina experiencing similar issues?
Are they experiencing the same network issues?
I don't know what the current situation is like there. However, I have a question. I don't live in the US, but I know hurricanes are common in the US.
I'm sure network outages can happen to other data centers. However, how long does it usually take for carrier partners to restore connectivity to a data center in a disaster situation like this?
Each country has different capabilities in terms of recovery. I certainly can't equate network recovery times in developing countries and in the US.
maybe this can give me an idea of how long the recovery process can take.
Is restoring connectivity to a data center a priority for carrier partners over regular consumers?
Yes.
No, every situation is unique.
As soon as possible.
Hours to months.
Yes.
This is one of those it won't be resolved until it's resolved situations. We had something similar in the past where both redundant lines in a dark fiber were cut and we introduced a second dark fiber ring to a diverse physical location to prevent it from happening again.
This is why I told Manny at HostDime (first employee) back 20 years ago it was stupid to put a datacenter in Orlando.
That summer there were 3 hurricanes.
After many outages and heat issues, quickly went from a decomissioned UUNet data center in downtown Orlando, where we weren't able to get HVAC installed (too heavy for roof) to a decomissioned aircraft hangar that's just outside of Orlando in Maitland, across from a Popeyes at the time.
Built the entire thing out from the ground up and got TWTelecom. I even coined the phrase DimeNOC, and that was their original DC. What saved them in the future from storms was TWTelecom was able to run fiber + single floor aircraft hangar.
The huge issue with a place like Lenoir, NC as @qps mentioned is back-haul to PoPs that are not even in the same state. Your data center could be humming along with the best diesel generators, but if your fiber is cut, or your transit provider didn't really think things out disaster recovery wise, you're dead in the water.
Hoping they get it solved quick, this is not fun for anyone.
He dosen't seem to have cared a lot though
This was early 2000's and DimeNOC 1 is still there AFAIK in Maitland. Looks like Popeyes is now a tire shop.
It was a great build out and they've come a long way.
He was helpful getting on an IX in Brazil at another company.
Our nodes are up now.
Me too.
We sincerely apologize for the delay in updating LET. Our data connections in the region were incredibly weak throughout yesterday and continuing today. To update everyone.
We are pleased to inform you that connectivity has now been restored as of this morning at roughly 2:30 AM EST. We deeply apologize for the inconvenience caused by the recent service disruption. Our region as many of you have acknowledged was severely impacted by Hurricane Helene, resulting in the loss of all four of our diverse carrier links.
We can also report that at no point during the storm did the facility lose power as all power redundancies worked as designed. I was on site through the storm and can also report that the facility itself is undamaged.
If you are still experiencing any issues, please don’t hesitate to open a ticket with our support team at [email protected].
We sincerely appreciate your patience and understanding during this challenging time.