KRYPT @ LAX10 DC issue

neilmendoza

Hi Guys,

I have receive below notification for many of my servers. Does anyone know why there are moving the servers when it is just power outage. The reason doesn't fit the action. Anyone's servers have the impact ?

We are experiencing an outage at our LAX10 facility that is affecting your
service. Evocative Data Center leadership is on site and is working to restore
service for our Krypt and ION customers. We have opened a ticket for your
account and we will be providing regular updates and remediation plans as the
situation progresses. There is currently no ETA for service restoration and we
will update you when that is available. We appreciate your patience, and if you
have immediate concerns or any questions, please contact us at
[email protected] or call 888-365-2656.

As of May/25/2023 - 9.00AM PDT – Our data center operations team and the network engineering team are on site at the facility. We are coordinating migration planning and hardware replacement and are working to establish an ETA for restoration of services.

Over the past 24 hours we have acquired new hardware to restore service to Krypt customers. We have started a migration process for the data on the affected servers to the new hardware. The cloud services team is continuing the work to restore service. Please contact [email protected] or call 888-365-2656 with any immediate concerns or questions you may have.

Thank you

Neoon

Prob. they can't fix it in a timely manner.

h2o

You should check this out.
https://www.reddit.com/r/sysadmin/comments/13o44on/comment/jlr415r/

As someone who has been in Colo4 before and after the fire it was surprising.

The cab that was on fire looks like the aftermath of a car bomb. The backs of the servers in there are melted and the doors to the cab are warped from the heat. The cable trays above the cab are completely missing from melting.

Digital reality didn't let evocative into colo4 until Tuesday morning, which is when they were able to start pulling critical gear out.

Water damage from what I was able to see was limited to row 4 and the rows next to it. > We got lucky as our that was near by did not get hit with water.

Most gear got soot in them however. The power went out before the sprinklers kicked on so systems that didn't take direct water needs to be cleaned out of soot, gear that got hit needs to be cleaned of soot and water debris or be replaced.

There are teams of people cleaning colo4 of the damage as well as teams of people pulling hardware to replace or clean / repair.

Its hard to wrap the mind around the amount of work still needing to be done. Especially for all of the networking gear.`

neilmendoza

Thank you and I saw that thread @h2o , there is no official news from them.

aveline

@neilmendoza said:
Thank you and I saw that thread @h2o , there is no official news from them.

https://www.datacenterdynamics.com/en/news/fire-hits-digital-realty-facility-in-el-segundo-california/

lowendclient

No it's not only power outage...