New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Hello, sir. The network has been down for about 30 minutes.
Updates on X.
https://x.com/Oplinknet/status/1810766223833243803
We are posting updates on x: https://x.com/oplinknet
Sadly dealing with a double outage from both Lumen + Cogent at this time due to the hurricane beryl aftermath. Both 3rd party companies are having backup power issues at their DCs that provide Oplink with internet.
can only blame on god at this point
Or ERCOT, and their entire joke of an electric system.
Their 'grid' can't handle weather, and since they're disconnected from the US grid, you'll have days you end up paying 1,600% more per kWh due to deregulation, or just have the east of the state without power.
Feel bad for Oplink, completely out of their hands as it's uplink, but Texas is not where you want production servers.
I don't suppose anybody knows, but I'll ask anyway.
Any ETA for a resolution with this?
If this is going to be a long-term outage then I need to get the disaster recovery process started for our affected servers.
No ETR at this time. We hope sometime this evening. Our guess is Lumen will come back first as they have been working on their issues the longest.
Do you have any updates from Cogent or Lumen that you can share?
Sadly, I just moved the server to oplink for a few hours and immediately encountered this problem, will I submit a ticket and will they refund me?
From what I've heard they've had no electricity since the hurricane beryl aftermath, and now everything died because the generators ran out of fuel.
oops, bad news.
The downtime is because of a natural disaster in Texas couple days ago. General concensus says the host is still good. You should give it some time and let it play out.
If this does not work, feel free to take your business elsewhere. But rest assured, there is no one except you, your backups that can protect you from all.
Any and all providers, big and small can face downtime.
Agreed. And this outage is from a direct hit of a hurricane. There are >2.5 million people out of power in the Houston area according to the news. I'd challenge any hosting provider to have perfect uptime through a direct hurricane blast. So this is definitely "act of god" territory. Let's see how they handle it by tomorrow and the next day.
Now my only slight criticism -- and maybe @oplink can reply to this -- is that they don't seem to have an offsite status page, other than their Twitter/X feed. So perhaps they can pay for an external service outside of Texas for such a service where we can get better/comprehensive reports. If they have such a service, I haven't seen it, and I apologize for missing it.
Otherwise, I think Oplink should be given a pass on this so far. In fact, they actually stayed up during the brutal frontal attack of the hurricane, which was great. They lost Lumen, but Cogent kept on ticking... then the storm passed, generators started running out of fuel, the whole city is a mess, millions of people out of power, and finally Cogent went down.
I think what we should be asking is if @oplink and their staff and family members are healthy and safe? Crossing fingers all is well with them.
Let me be 100% clear here, the Oplink.net datacenter where all our clients Colo, dedicated servers, virtual servers, and virtual dedicated servers are located in has has had zero issues with power/cooling. All devices and servers are still booted and should simply just reconnect to the network once its back online!
Update as of 6.34PM Central:
Lumen has restored power and cooling to their datacenter where our Fiber runs to. There are cards that ran into overheating failures during the power outage. Once these are operational again, Lumen should come back online for everyone.
Cogent, our other carrier had its own issues w/ power and we are still waiting for a detailed outline of what failed. From what we know the Cogent datacenter had 2 backup generators. One failed sometime during the storm. The faciality was running on 1 working generator. That working generator failed for some reason, Cogent is not sharing any further details at this time.
Sadly our two Internet Backbone providers and they both failed us during this storm/event. This was totally out of our control.
Thank you for your understanding.
Time to hook up Starlink/5G modem to the DC switch
im also a customer of oplink
if oplink has only 1 upstream then I would complain that it is provider fault a little bit
but this, 2 upstreams gone ded, I understand the stress they are going through rn
Impressive. You guys took a direct hit. Sorry to hear about the two providers though, but sounds like they'll be back online soon. Hope you and yours are safe and sound.
get a single t-mobile 4g modem 30gb $10 a month plan to run off of for entire DC backup.
Texas powergrid is kind of a meme lol. I wonder how the DC handles surge pricing when it hits thousands per kwh peaks or whatever BS.
I migrated a VPS from Texas (Other Provider) through Oplink yesterday, to Columbus.
Now the Hurricane is heading to Columbus.
It's chasing me....
It looks like cogent is up
I think a helpful piece of information would be, is Cogent up on the power grid or on generators?
Because if it's generators we've seen what happens when they run out of fuel and that may still happen. So disaster recovery processes may need to keep going.
Was fun while it lasted
Cogent came up and bounced once but is holding. They are still on generator power.
Lumen is still down. They say A lot of their hardware overheated and tossed alarms in some of the cards/modules. We expect them to come back sometime soon as they had tons of replacements delivered to the site. We feel like they’re getting close.
How long the hurricane existed ?
It came and went through Houston all within ~12hrs however had ~100 MPH/160KPH winds with it.
I think their downtime was remarkably short considering that as of this morning in their area, there are still about 1.5 million people without power. I have a friend in the area that had a tree fall on their house (they are fine, thank goodness).
As for downtime, maybe one can blame Texas for their power grid, and maybe @oplink could have had a third network provider for another backup, but I'm not sure that would have helped with a direct hit of a storm that size.
In any case, I do hope they add an external status page instead of just their Twitter/X updates.
Hopefully no more bouncing or power problems today.
And BTW, this also highlights once again the importance of having a good disaster plan with ANY provider. If you are running mission-critical sites, then you'll want to have appropriate redundancy and policies in place, because anything can happen anywhere to anyone and to any provider.
Oplink Team Update:
Our team is safe, some without power still at our homes.
Network Update 6.57AM:
I am told the Lumen DC is now fully on 2 portable generators and all cooling in the facility is operational. They are working with clients at this point. They are bringing a 3rd portable generator in from New Mexico for precautions.
Lumen should be up shortly, they had to replace our 100G card in their DC that may have overheated. Routes might blip 20-30secs when Lumen comes back up.
We expect Lumen to come up in 1-2hrs if not sooner.
Adding a third carrier may have helped. Most were down. The only one I heard that remained up the entire time was Crown Castle. It would be like playing the Backbone lottery 1+yrs in advanced. Whos to say they will be up during the next outage? Do we add a 3rd? a 4th? So much for keeping lower prices. This is 100% an ultra rare event. I am positive our two uplinks will implement changes and upgrades to prevent repeat issues like this. With that said I don't see Oplink adding more Uplink's at this time. We have never had a double uplink outage like this in 20yrs of being in business. Everyone major company even Amazon has outages, no matter how much redundancy is added.
We have been and will remain a very reliable provider.
Thanks for everyone's understanding during all this.
Thank you for your business!
Regards,
Ryan