New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
The fire department's inspection is scheduled to begin in just over an hour (9AM EST).
Keeping my fingers crossed they're happy with the state of the cleanup, and safety systems.
This is an opportunity to undertake a full fire safety inspection. If the inspector finds a small bag of trash temporarily dropped in front of the emergency back door, the inspection fails with a fine.
We greatly respect the work of these professionals who give their lives for other people, but it is unbelievable that they do not take into account the financial damage that is being generated. We are from Brazil and we are not proud of the problems we have here, but in situations like these we need to have an understanding of what it takes to be agile and try to understand that there are no efforts by anyone to return the activities of the companies and people affected because these are that pay taxes, generate jobs and are a means of maintaining economic and social life. As an example of the friend in the last post, a bag preventing this whole operation from working? This building has over 15 floors. It houses hundreds of datacenters and we are at the mercy of one person or group of people who can reliably use text interpretation to assert the famous who is in charge.
While I too am frustrated because the site I manage is down because of this, I am not frustrated with the fire marshal doing his job diligently. I am frustrated with the people who did not maintain this system well enough to prevent the fire or, if the fire couldn't have been prevented, to quickly pass any inspection because the systems were kept in passing condition. The solution isn't to relax safety standards; it is to increase preventative maintenance.
If you have serious financial damage from one location being down, you should have redundancy in place. These days, it's not terribly difficult, even with open source software like Xen Orchestra or commercial solutions like Zerto can mirror data off-site easily.
Agreed. The fire marshal is doing his job so that once the DC is powered up again, it is safe for all the people going in and out.
And if the DC management thinks that the fire marshal is abusing his power and causing needless delays, then they can sue the fire marshal in a court of law. They have the money and the lawyers. But I doubt that would get very far, because from what I'm reading, the place wasn't in good shape. Also keep in mind we are getting updates from a biased source - the DC. Now it would be great if we had full transparency, and read all the full reports from the fire marshal too, and his after action report. I'll bet the language would be a little different than what the DC is saying.
In any case, the hyper-optimistic first take from the DC was clearly way off base from realistic safety standards and compliance, and that should give every provider who uses that DC something to think about.
I feel bad for the all the providers hosted there, and all the disruption. I've got stuff there too. I hate this downtime. But I already implemented my contingency plan. Anyone who has anything valuable should ALWAYS have a contingency plan for unfortunate events like this.
Latest update:
I heard the preliminary inspection is good and we are taking steps to energize the property now.
I’m waiting for the official update from DC Ops. More to come.
P.S. And BTW, for those not in the US, you might not be aware that there are rigorous standards of safety that ANY business that operates here has to comply with. Some people ignore them or don't comply with them, but when an accident happens because of lack of safety compliance... well, let's just say the law suits will start flying.
There are also regional/state regulations, and other regulations that the DC would have already been aware of. Just starting with OSHA regulations and so forth. Right or wrong, too much or too little, that's the situation in the US. So while people are complaining about the fire marshal, he has to comply with his duty to uphold safety standards, but also keep in mind that all those regulations are NOT a mystery to the DC management. They are supposed to know the regs and comply. If someone had gotten hurt in this situation (thank goodness it looks like no one was hurt!), then this would be a different situation and the DC would be down even longer.
My impression is an automatic Halon extinguishing system flooded the data room with extinguishing agent as these systems do, and the cleanup involved removing the chemical from the entire room which was likely an inch thick on all the equipment and floors in the room. Then after they cleaned it all up, the Fire Marshall insisted that the Automatic Halon fire suppression system be completely re-installed being that it was triggered and expended.
Did you hear that from your rep? I only got a generic update from evocative that they are still meeting and its still powered down
Do you mean electrical room (where the issue occurred)?
You might be thinking of a different datacenter. This one is not a high-rise with 15 floors. It's shaped more like a warehouse, and I believe it has two floors.
https://www.google.com/maps/place/1+Enterprise+Ave+N,+Secaucus,+NJ+07094/
Thanks. We are not on site and we do not have concrete information such as photos and more details. Any updates on the meeting with fire marshal?
Latest from Evocative:
Dear Valued Customers,
Our site inspection this morning went well and we have been granted authorization to restore utility power to the site and are currently working on re-energizing utility power to the facility. Our onsite team is working with the fire marshal and electrical inspectors, ensuring electrical system safety as we prepare to bring utility power back to the site.
Once that is completed, we will work towards bringing up our critical infrastructure systems. This will take approximately 5 hours.
While we are working on that, we will also be working on our fire/life safety systems as we need to replace some smoke detectors and have a full inspection of the fire system prior to allowing customers to enter the facility.
We will be sending out hourly updates as we make progress on bringing the facility back online.
Is there any link where it is possible to follow this hourly status?
The first line of the very first message in this topic contains the link.
Hopefully they get the critical infra running and it doesn't fail if they try to do so.
Everything still offline
Looks like the problems in the background is much bigger than I (we?) thought it, if they was not able to solve/handle it until at this time. Or the problems much bigger than they told us.
Latest updates from DediPath: https://i.imgur.com/61dOTbV.png
Yeah, that's the next step. It also sounds like they had some issues with basic fire safety things, so hopefully those can be fixed easily.
Of course, when everything is powered up, the various tenants will have to deal with the fact that their hardware suffered two hard power off events. There will inevitably be failures of various equipment like power supplies and hard drives.
And that doesn't take into account that the datacenter temperature has most likely been extremely high since the shutdown. It's supposed to be 93° F (34° C) today in Secaucus, and yesterday was not much cooler. I suppose most hardware will have been okay, but I guarantee it's not very nice for humans.
I did the same thing I trusted one location when in reality you should always have a backup location this is not Kamatera fault it's the data center or even the battery that failed if that is the case. You should always have a way to access your data. I do hope you back every thing up at a different location but I don't think it's fault we don't know the location what is next to it. I am sorry you have to wait as I do I to don't have a backup I run a email server that I don't have a second server elsewhere I will be making a second server elsewhere since this happened it light a little fire under me. Best of luck
Latest:
Dear Valued Customers,
We have completed the full site inspection with the fire marshal and the electrical inspector and utility power has been restored to the site.
We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site. Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn up.
The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
We are planning for a late afternoon/early evening time frame when clients will be able to come back on site.
We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
Yes, electrical room, data room, whichever. I've seen these systems in both and they make a huge mess when they get triggered, EVERYTHING in the room needs to be cleaned up. But this statement concerns me: "fire system components replaced" which means he likely wants the entire system functional again regardless of whether it's required or not.
In a case like this, would it be reasonable to expect some kind of refund from my VPS host (virmach) for the down-time?
All these inspection, cleanup, …, are taking too long.
Why not rope it off and rebuild one town over?
You check TOS, you check SLA and act on that.
Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.
As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.
We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
Yikes. Almost every single announcement from the DC so far has been on the super-optimistic side of the equation, bordering on wishful thinking.
@EthernetServers Antecipadamente agradeço as informações prestadas. Como nosso fornecedor (Kamatera) não esta nos atualizando como deveria sua ajuda esta sendo fundamental. Não haveria uma atualização a cada 1 hora? Pergunto pois esta conduta está ajudando a mitigar a insatisfação de nossos clientes.
@EthernetServers Thanks in advance for the information provided. As our supplier (Kamatera) is not updating us as it should, your help is being fundamental. Wouldn't there be an update every 1 hour? I ask because this conduct is helping to mitigate the dissatisfaction of our customers.