krypt has been down for more than ten hours

suut

The source said that the cause of the accident was a fire in the computer room, like ovh.

hkbbdx

my vps too

louiejordan

It's not a very important provider, and many people haven't even heard of it.

yanling

the website is down now https://www.krypt.com/

Nanja

@louiejordan said:
It's not a very important provider, and many people haven't even heard of it.

You are right, I have never heard of them. They aren't important to me.

The companies revenue is more than $15 million. The Asia branch is worth more than $5 million.

I imagine they are important to someone with that kind of cash coming in.

Interesting thing I discovered is that Asia site is down also...

They couldn't have burned down both places...

suut

@Nanja said:

@louiejordan said:
It's not a very important provider, and many people haven't even heard of it.

You are right, I have never heard of them. They aren't important to me.

The companies revenue is more than $15 million. The Asia branch is worth more than $5 million.

I imagine they are important to someone with that kind of cash coming in.

Interesting thing I discovered is that Asia site is down also...

They couldn't have burned down both places...

krypt was created longer than ovh.

Nanja

From the company that owns Krypt I guess.

We are experiencing an outage at our LAX10 facility that is affecting your service.
Evocative Data Center leadership is on-site and is working to restore service for our Krypt and ION customers.
We will be providing regular updates and remediation plans as the situation progresses.
There is currently no ETA for service restoration and we will update you when that is available.
We appreciate your patience, and if you have immediate concerns or any questions, please contact us at [email protected] or call 888-365-2656.

bdl

.

Neoon

h2o

It has been more than 24 hours.

JabJab

Pictures?! Any (more specifc) news?

That LAX10 DC seems to be located in some warehouse/buisness district of LA - there is no way no one have pictures if shit on fire

malignify

Krap.

WebProject

@JabJab said:
Pictures?! Any (more specifc) news?

That LAX10 DC seems to be located in some warehouse/buisness district of LA - there is no way no one have pictures if shit on fire

Here is pic:

No issues reported probably just network issue.

suut

@WebProject said:

@JabJab said:
Pictures?! Any (more specifc) news?

That LAX10 DC seems to be located in some warehouse/buisness district of LA - there is no way no one have pictures if shit on fire

Here is pic:

No issues reported probably just network issue.

The customer service claimed that there was a fire, but there was no further fire announcement.

MannDude

Fire doesn't automatically mean a building engulfed in an inferno.

It's LA. They're going to have strict fire codes. Fire could have broken out and been quickly suppressed, but power could and likely is cut, I'd imagine.

Neoon

@MannDude said:
Fire doesn't automatically mean a building engulfed in an inferno.

It's LA. They're going to have strict fire codes. Fire could have broken out and been quickly suppressed, but power could and likely is cut, I'd imagine.

Not if you floor is made out of wood.

Nanja

I tried talking to the them about the fire and asked if everyone lost their data. I was curious about what happened. They won't give me any answers in regard to it and the question got avoided. They just said that I must be a customer and open a ticket to get updates about the outage. Why would they avoid my question about the fire?

Is it because of a legal reasons that they can't disclose that fact?
They could've just said hey... sorry to tell you, all our servers burnt up, but instead said open a ticket if you are a customer.

I don't see any information about it in any social media, why are they so hush hush about this fire.

qps

https://www.reddit.com/r/sysadmin/comments/13o44on/digital_realty_el_segundo_fire_5212023/

niuwo

remark , still down now

Francisco

@qps said:
https://www.reddit.com/r/sysadmin/comments/13o44on/digital_realty_el_segundo_fire_5212023/

Ok, who was overclocking Ryzen 7900x3ds?

Francisco

Ganonk

GWS™

febryanvaldo

@Neoon said: Not if you floor is made out of wood.

I'm pretty sure you are reffering to someone

LaurelNotYanny

Still, there has not been any official statement released regarding the outage from iON or Krypt?

CheepCluck

The offical statement is to standby patiently for the official statement.

GamingAge

Since this forum thread and that subreddit seems to be the only sources of "up-to-date" info for this mess, the latest ticket update from Krypt (have a web server there) we received was...

As of May 23, 7.00pm PDT, Evocative DC Operations has access to the facility. We
are coordinating migration planning and hardware replacement in order to restore
service. Currently no update on ETA for restoration of services.

suut

Dear valued customer,

As previously shared, our Los Angeles LAX10 colocation suite (“Colo 4”) at 2260 W El Segundo Blvd was affected by an event due to causes beyond our reasonable control on Sunday, May 21, 2023, resulting in fire and water damage rendering Colo 4 inoperative. In response, we expedited the installation of new cabinets, power circuits, and network fiber connectivity in an adjacent colocation suite within the same building, reducing the standard installation process by several weeks.

Our colocation, cloud, and bare metal customers in Colo 4 were directly affected by this incident. As of today, colocation customers have been provided the infrastructure necessary to migrate their environments, cloud customers with offsite disaster recovery services have been restored at a remote facility, and bare metal server environments are being moved to a new location for data access. Unfortunately, our local cloud storage and backup systems in Colo 4 were significantly affected by the incident and cloud customers without offsite disaster recovery services are still affected. We have replaced hardware for both the local storage and backup systems, but several hard drives remain damaged, affecting data recovery efforts. We are actively working with drive manufacturers on data recovery solutions.

We appreciate your continued patience and will provide further updates to impacted customers as we progress. If have any immediate concerns or additional questions, please contact us at [email protected] or call 888-365-2656.

treesmokah

@suut said:

Dear valued customer,

As previously shared, our Los Angeles LAX10 colocation suite (“Colo 4”) at 2260 W El Segundo Blvd was affected by an event due to causes beyond our reasonable control on Sunday, May 21, 2023, resulting in fire and water damage rendering Colo 4 inoperative. In response, we expedited the installation of new cabinets, power circuits, and network fiber connectivity in an adjacent colocation suite within the same building, reducing the standard installation process by several weeks.

Our colocation, cloud, and bare metal customers in Colo 4 were directly affected by this incident. As of today, colocation customers have been provided the infrastructure necessary to migrate their environments, cloud customers with offsite disaster recovery services have been restored at a remote facility, and bare metal server environments are being moved to a new location for data access. Unfortunately, our local cloud storage and backup systems in Colo 4 were significantly affected by the incident and cloud customers without offsite disaster recovery services are still affected. We have replaced hardware for both the local storage and backup systems, but several hard drives remain damaged, affecting data recovery efforts. We are actively working with drive manufacturers on data recovery solutions.

We appreciate your continued patience and will provide further updates to impacted customers as we progress. If have any immediate concerns or additional questions, please contact us at [email protected] or call 888-365-2656.

Reasonable response, i would not be angry with how they are handling it.
I wish they told us how it happened, but i do not think its a "must".

jbiloh

Feel badly for everyone impacted by the fire at that facility. So unfortunate that a single device in a single cabinet has caused such a big inconvenience.

LaurelNotYanny

I received this as well, from [email protected]

@suut said:

Dear valued customer,

As previously shared, our Los Angeles LAX10 colocation suite (“Colo 4”) at 2260 W El Segundo Blvd was affected by an event due to causes beyond our reasonable control on Sunday, May 21, 2023, resulting in fire and water damage rendering Colo 4 inoperative. In response, we expedited the installation of new cabinets, power circuits, and network fiber connectivity in an adjacent colocation suite within the same building, reducing the standard installation process by several weeks.

Our colocation, cloud, and bare metal customers in Colo 4 were directly affected by this incident. As of today, colocation customers have been provided the infrastructure necessary to migrate their environments, cloud customers with offsite disaster recovery services have been restored at a remote facility, and bare metal server environments are being moved to a new location for data access. Unfortunately, our local cloud storage and backup systems in Colo 4 were significantly affected by the incident and cloud customers without offsite disaster recovery services are still affected. We have replaced hardware for both the local storage and backup systems, but several hard drives remain damaged, affecting data recovery efforts. We are actively working with drive manufacturers on data recovery solutions.

We appreciate your continued patience and will provide further updates to impacted customers as we progress. If have any immediate concerns or additional questions, please contact us at [email protected] or call 888-365-2656.

emg

What fuels those fires in a colocation facility like that? Could there be quantities of lithium batteries there? (I assume they use lithium batteries in UPSs, right?)

Robert_baudcom

so long time, have not recovered. I worry much about the data. Do not know they have backup or not.