Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Carrier1/Evocative/Prime (1515 Round Table, Dallas, TX) Power Outage
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Carrier1/Evocative/Prime (1515 Round Table, Dallas, TX) Power Outage

jonbeardjonbeard Member, Patron Provider
edited May 26 in Outages

Looks like Carrier1/Evocative/Prime Datacenters in Dallas TX is having a total blackout outage. We havent heard from the facility but suspect this is the case. All of our prelim network tests point to a power outage at the facility. Apparently there was a storm that just ran threw there. A few days ago, they just tested generators/ATS functionality... Not sure how we go to this...

Just wanted to give folks a heads up, as this is affecting everyone in both of their facilities.

Thanked by 3MannDude balramm themew
«13

Comments

  • balrammbalramm Member, Host Rep

    I can confirm all our infra is down in Dallas.

  • balrammbalramm Member, Host Rep
    edited May 26

    @balramm said:
    I can confirm all our infra is down in Dallas.

    Also got an email from Cogent Dallas

  • wdmgwdmg Member, LIR

    @jonbeard said: A few days ago, they just tested generators/ATS functionality

    Could you define the test cases? "Looks clean, I like how it looks. Let's say it passed."

  • jonbeardjonbeard Member, Patron Provider
    edited May 26

    @wdmg said:

    @jonbeard said: A few days ago, they just tested generators/ATS functionality

    Could you define the test cases? "Looks clean, I like how it looks. Let's say it passed."

    Evocative blend gateways are totally down. All IP's past the MMR have stopped announcing. MTRs never reach the facility. Also we have folks onsite that have confirmed the lights are on (emergency) but everything in both datacenters are totally silent and dark. All doors are unlocked (mag stripes) which tells me even their backbone has went dark.

    In my experience of datacenter technician work, this sounds like an ATS failure. Hivelocity and all other carriers in the building are offline. Cogent has also emailed us the same email as prior user indicated as well.

    Bad storms rolled through and I suspect this is the cause.

  • jordijordi Member, Patron Provider

    hivelocity:

    Our Dallas facility provider, Prime(formerly Evocative and Carrier-1), has lost utility power. Initially the power load was transferred to battery backup and then generator as designed. However, at some point in time the generators dropped the critical load. Prime has electrical engineers investigating the problem and the solution. We will update you all as things progress.

    We are also waiting for the solution, all the servers are down

  • jonbeardjonbeard Member, Patron Provider

    Does anyone have confirmation that utility power was restored quite some time ago? I suspect a failure within their ATS, but thats just my 2 cents without being on site.

    Evocative still has not replied to any of our inquiries. No ETA's are being provided to people who are onsite.

  • wdmgwdmg Member, LIR

    @jonbeard said:
    Does anyone have confirmation that utility power was restored quite some time ago? I suspect a failure within their ATS, but thats just my 2 cents without being on site.

    Evocative still has not replied to any of our inquiries. No ETA's are being provided to people who are onsite.

    Last ETA I heard was 30-60min from a friend in the DC, that was 45 minutes ago though.

  • DPDP Administrator, The Domain Guy

    Will pin this for now.

  • jordijordi Member, Patron Provider

    It's already been +2 hours waiting, there is no confirmed ETA, they only tell us that the technicians are on site

  • Just got an update saying: [email protected]

    8:28 AM (6 minutes ago)

    to me
    We're currently working on getting power fully restored after a power outage.
    No ETA as of yet.

  • I also asked why generator didn't cut over and apparently there was some issue with that and the vendors are on site trying to get that going. No clue on the power restoration.

  • sohailsohail Member

    Estimated restoration, 12 p.m. Not looking good. I was informed by my provider that they have on site technicians 45 min ago.

  • Hopefully they can get the generators back up and running. That would be a big relief.

  • jordijordi Member, Patron Provider

    @sohail said:

    Estimated restoration, 12 p.m. Not looking good. I was informed by my provider that they have on site technicians 45 min ago.

    I just called hivelocity, they told us about 30 minutes. We hope it is true because 2 hours have already passed and it is unfortunate to be in this situation.

  • jordijordi Member, Patron Provider

    @sohail said:

    Estimated restoration, 12 p.m. Not looking good. I was informed by my provider that they have on site technicians 45 min ago.

    I just called hivelocity, they told us about 30 minutes. We hope it is true because 2 hours have already passed and it is unfortunate to be in this situation.

  • RadiRadi Host Rep, Veteran

    Hopefully it won't be too long.

  • Power seems to be getting restored, our infra in Dallas has power now

  • PureVoltagePureVoltage Member, Patron Provider

    Still awaiting updates from the DC both A+B power failures they messed up something I would suggest anyone to restore any critical services to another location. Who knows how much longer it will end up being.

  • I just called them and they say power has been restored - but they are powering on a little at a time so they don't overload.

  • jordijordi Member, Patron Provider

    For our part, the servers are online

  • PureVoltagePureVoltage Member, Patron Provider

    show system uptime
    up 35 mins

    They finally have most things coming back online.
    Took a while I assume to get all of the MMR online.

  • balrammbalramm Member, Host Rep

    Everything is now restored for us as well.
    As per our monitoring- Downtime: 2 hr 57 min

  • Good to hear, hopefully our stuff is coming back online too.

  • HivelocityHivelocity Member, Patron Provider

    We are beginning to see power restoration to our servers at DAL1, inside Prime. We have technicians on site going from cabinet to cabinet working to get every server back online ASAP.

  • @Hivelocity I am assuming they have to power on in phases, correct? To prevent a major spike in power that could trip circuits?

  • HivelocityHivelocity Member, Patron Provider

    @zipservers said:
    @Hivelocity I am assuming they have to power on in phases, correct? To prevent a major spike in power that could trip circuits?

    Correct, you do not want to restore power to every server at once under a scenario like this for the reason you just mentioned.

  • cubedatacubedata Member, Patron Provider
    edited May 26

    CubeData's infrastructure has been down since 7am Eastern Time Zone according to HetrixTools Monitoring.

    So sadly everything is down at the moment, I only found out this was related a power outage because I contacted Psychz Networks Remote Hands thinking the Firewall went down and just needed to be rebooted and they said nope, it wasn't that, it was related to a power outage in Dallas.

    This kinda reminds me of the waiting game long ago with Delimiter when I had servers with them before.

  • jonbeardjonbeard Member, Patron Provider

    @Hivelocity Did you go out there? In the Range Rover, Porsche, Corvette or Bentley? :wink: You always know when theres a big outage because the nice cars show up haha

    Happy Sunday!

  • zeeter82zeeter82 Member

    Looks like my Dallas VPS with HostUS is down - I'm assuming it's part of this outage.

Sign In or Register to comment.