New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Carrier1/Evocative/Prime (1515 Round Table, Dallas, TX) Power Outage
Looks like Carrier1/Evocative/Prime Datacenters in Dallas TX is having a total blackout outage. We havent heard from the facility but suspect this is the case. All of our prelim network tests point to a power outage at the facility. Apparently there was a storm that just ran threw there. A few days ago, they just tested generators/ATS functionality... Not sure how we go to this...
Just wanted to give folks a heads up, as this is affecting everyone in both of their facilities.
Comments
I can confirm all our infra is down in Dallas.
Also got an email from Cogent Dallas

Could you define the test cases? "Looks clean, I like how it looks. Let's say it passed."
Evocative blend gateways are totally down. All IP's past the MMR have stopped announcing. MTRs never reach the facility. Also we have folks onsite that have confirmed the lights are on (emergency) but everything in both datacenters are totally silent and dark. All doors are unlocked (mag stripes) which tells me even their backbone has went dark.
In my experience of datacenter technician work, this sounds like an ATS failure. Hivelocity and all other carriers in the building are offline. Cogent has also emailed us the same email as prior user indicated as well.
Bad storms rolled through and I suspect this is the cause.
hivelocity:
Our Dallas facility provider, Prime(formerly Evocative and Carrier-1), has lost utility power. Initially the power load was transferred to battery backup and then generator as designed. However, at some point in time the generators dropped the critical load. Prime has electrical engineers investigating the problem and the solution. We will update you all as things progress.
We are also waiting for the solution, all the servers are down
Does anyone have confirmation that utility power was restored quite some time ago? I suspect a failure within their ATS, but thats just my 2 cents without being on site.
Evocative still has not replied to any of our inquiries. No ETA's are being provided to people who are onsite.
Last ETA I heard was 30-60min from a friend in the DC, that was 45 minutes ago though.
Will pin this for now.
It's already been +2 hours waiting, there is no confirmed ETA, they only tell us that the technicians are on site
Just got an update saying: [email protected]
8:28 AM (6 minutes ago)
to me
We're currently working on getting power fully restored after a power outage.
No ETA as of yet.
I also asked why generator didn't cut over and apparently there was some issue with that and the vendors are on site trying to get that going. No clue on the power restoration.
https://stormcenter.oncor.com/
Estimated restoration, 12 p.m. Not looking good. I was informed by my provider that they have on site technicians 45 min ago.
Hopefully they can get the generators back up and running. That would be a big relief.
I just called hivelocity, they told us about 30 minutes. We hope it is true because 2 hours have already passed and it is unfortunate to be in this situation.
I just called hivelocity, they told us about 30 minutes. We hope it is true because 2 hours have already passed and it is unfortunate to be in this situation.
Hopefully it won't be too long.
Power seems to be getting restored, our infra in Dallas has power now
Still awaiting updates from the DC both A+B power failures they messed up something I would suggest anyone to restore any critical services to another location. Who knows how much longer it will end up being.
I just called them and they say power has been restored - but they are powering on a little at a time so they don't overload.
For our part, the servers are online
show system uptime
up 35 mins
They finally have most things coming back online.
Took a while I assume to get all of the MMR online.
Everything is now restored for us as well.
As per our monitoring- Downtime: 2 hr 57 min
Good to hear, hopefully our stuff is coming back online too.
We are beginning to see power restoration to our servers at DAL1, inside Prime. We have technicians on site going from cabinet to cabinet working to get every server back online ASAP.
@Hivelocity I am assuming they have to power on in phases, correct? To prevent a major spike in power that could trip circuits?
Correct, you do not want to restore power to every server at once under a scenario like this for the reason you just mentioned.
CubeData's infrastructure has been down since 7am Eastern Time Zone according to HetrixTools Monitoring.
So sadly everything is down at the moment, I only found out this was related a power outage because I contacted Psychz Networks Remote Hands thinking the Firewall went down and just needed to be rebooted and they said nope, it wasn't that, it was related to a power outage in Dallas.
This kinda reminds me of the waiting game long ago with Delimiter when I had servers with them before.
@Hivelocity Did you go out there? In the Range Rover, Porsche, Corvette or Bentley?
You always know when theres a big outage because the nice cars show up haha
Happy Sunday!
Looks like my Dallas VPS with HostUS is down - I'm assuming it's part of this outage.