millenial.host (mlnl.host) | Deadpooling or down?

randvegeta

@fluffernutter said: Do you own the building? Was there not any hands staff available for a building wide outage?

No we don't own the building. In Hong Kong? Who can afford to own an entire building!

We (the data centre) own the unit (space) in the building where all the equipment is. But not the entire building. There were building staff on site (meaning security and management), but not OUR staff. We have staff who live nearby (15 mins walk or 5 mins by taxi).

Even if we had someone onsite, it's unlikely we would have been able to do much. We have a generator, but as mentioned, it's not something that we are comfortable running unattended. We ran it during the last power outage 6 months ago and there is a pretty sizable oil leak. Usually we have long outages once every 5 yrs (building maintenance) but this one wasn't scheduled. And since we had the scheduled maintenance in the last year, the expectation was that we would not need the generator any time soon. We made the decision to invest in batteries and UPS, rather than maintaining the generator. The generator cost a lot to run, maintain and just store. It occupies significant square footage of our space, and HK grid power is extremely reliable. Our existing UPS setup is intended for brownouts, rather than extended outages.

We plan to get the entire DC on 1 hour UPS supply as soon as possible. Our core can already handle 4+ hours. More or less we plan to add an extra half hour of battery capacity to our UPSs every month. By end of year we should have ~6hr run time for the whole DC. That's enough time to absorb the duration of every outage we've had for the last 15yrs! We will continue to build capacity until we have 24 hours. So about 4 yrs or so to do that.

fluffernutter

@randvegeta said:

@fluffernutter said: Do you own the building? Was there not any hands staff available for a building wide outage?

No we don't own the building. In Hong Kong? Who can afford to own an entire building!

We (the data centre) own the unit (space) in the building where all the equipment is. But not the entire building. There were building staff on site (meaning security and management), but not OUR staff. We have staff who live nearby (15 mins walk or 5 mins by taxi).

Even if we had someone onsite, it's unlikely we would have been able to do much. We have a generator, but as mentioned, it's not something that we are comfortable running unattended. We ran it during the last power outage 6 months ago and there is a pretty sizable oil leak. Usually we have long outages once every 5 yrs (building maintenance) but this one wasn't scheduled. And since we had the scheduled maintenance in the last year, the expectation was that we would not need the generator any time soon. We made the decision to invest in batteries and UPS, rather than maintaining the generator. The generator cost a lot to run, maintain and just store. It occupies significant square footage of our space, and HK grid power is extremely reliable. Our existing UPS setup is intended for brownouts, rather than extended outages.

We plan to get the entire DC on 1 hour UPS supply as soon as possible. Our core can already handle 4+ hours. More or less we plan to add an extra half hour of battery capacity to our UPSs every month. By end of year we should have ~6hr run time for the whole DC. That's enough time to absorb the duration of every outage we've had for the last 15yrs! We will continue to build capacity until we have 24 hours. So about 4 yrs or so to do that.

Yeah by "building" I meant the DC, not the full building. Didn't realize you were that big.