New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
At this point, it feels like either the update/news coming out is suppressed or there is something else going on.
I mean if the fire was contained to a single room, and did not spread beyond it.
Why the heck would a fire marshall not allow them to re-check everything and power up for greater than 24 hours.
Well, if it was in a main electrical room it likely damaged expensive things that needs to be replaced by a commercial electrician.
I can not speak of all their datacenters of course, but Microsoft are just finishing up two new datacenters not too far from where I live, and I can promise you that they spare no expenses.
Two identical dc's, maybe 30km apart. They've been building for 10 years and they are just closing in on finishing the first one right now. Everything built from absolute scratch, they bought empty land and started digging. We are talking 10's if not 100's of millions in each dc.
@EthernetServers, any updates? I know it's not in your control, but If you have any info you can share about an ETA on when they'll be back up it would help.
The latest update Evocative have given is:
Site management, the fire marshal, and electrical contractors are currently meeting to review the process of the cleaning effort to get approval from the fire marshal to re-energize the site.
We will update you as soon as the meeting has concluded.
Problem is this whole issue is most likely far greater than being let out. I don't know if anyone here has been onsite or not. If it was in fact their main UPS room that caught fire it will likely be down quite a bit longer due to the issues involved in this.
I have no idea how this data center was built but I know most UPS rooms require pretty heavy fire ratings. I know one we toured in the Seattle area years ago had A+B sides sectioned apart so one side could burn and keep things online using redundant power. It seems this isn't the case here.
If they got told today that they need more work to be done it will at least not go back online for another 16-18 hours I am assuming unless they can have the fire marshall come out after hours.
@Ernie do you know more about this outage? is there an ETA for when the servers will be up again?
I already posted the update from Dedipath about it here:
https://lowendtalk.com/discussion/comment/3687714/#Comment_3687714
My (not low-end) provider has been feeding similar updates as others shared above, with more details about a phased re-powering of the datacenter after being cleared by the fire marshal.
Update @ 9:15am EDT on 7/11: The facility has worked through the night cleaning equipment but has not yet reached the next stage of having it inspected. They are hoping for an afternoon re-energizing, but that process will also take a long time (many hours) as the facility has to be cooled as part of it, among other things.
Update @ 1:17pm EDT on 7/11: The cleaning continues, and an inspection is scheduled for 2pm EDT. If the fire marshal clears the site to be re-energized, they can start the process of re-energizing and testing their feeds and UPS equipment; over-cooling the facility; turning on other facility equipment and specific customer infrastructure (such as major backbone routers); then slowly bringing up customers in a staggered manner, being careful not to overload their power feeds. With all of this, they estimate that power may not be restored for another 6 hours.
TLDR; Their estimate is at least 7pm Eastern, and I'll share more updates that come.
I watched only the first and second series. The first was a masterpiece in creativity, every episode could have been a feature film in itself with lots of novelty and skills put into each episode, but the second series instead felt more like routine TV screenplay and was not as well thought out as the first one IMHO, despite the production budget was higher. Sequels typically decline in quality as they live off the fame gained by their predecessor.
Yeah, House of the Dragon is the second series. It's supposed to be a prequel.
It definitely seems milked.
If you read between the lines of the updates, it almost sounds like the datacenter is ready and willing to power up at any time, but the fire marshal is a stickler and won't let them. Or perhaps, like you suggested, the damage is worse than we're hearing, and they are using this as an excuse.
The constant mention of "cleaning" the room where the fire occurred is strange to me. I can certainly understand that a specialist company might need to be involved to clean up after a fire, but I don't know why this would involve the fire marshal.
As long as the UPS vendor has inspected the equipment and given it the okay, and maybe disabled the affected UPS and failed over to an alternate, then I don't see why a fire marshal would care about the details of the cleanup/remediation process.
Oh no I screwed over my terminology. I watched only the first two seasons, not series. TV is complicated these days... I should go back to coloring books
Apparently all they needed was a good cleaning:
"The facility has worked through the night cleaning equipment but has not yet reached the next stage of having it inspected."
"If the fire marshal clears the site to be re-energized, they can start the process of re-energizing and testing their feeds and UPS equipment"
As @srch07 pointed out, assuming the updates are accurate & there is no major damage - this has all the hallmarks of likely getting screwed over for not bribing going with the fire marshal's recommended cleaning provider
My thoughts exactly.
Feels like Marshal has a grudge with these guys, did they forgot to send him a greetings card?
Yet more cleaning, from the latest update:
This may never end...
Now, I am really curious. Was the UPS that dirty, that after 24 hours, they need to bring in more personnel to clean it with another 12 hours left, to meet deadline.
Update from my provider, nfoservers, with more details:
Update @ 4:41pm EDT on 7/11: We have received the following update from the facility, telling us that it will be at least another full day before our gear can be turned back on.
We have just finished the meeting with the fire marshal, electrical inspectors, and our onsite management. We have made great progress cleaning and after reviewing it with the fire marshal, they have asked us to clean additional spaces and they have also asked us to replace some components of the fire system. They have set a time to come back and review these requests at 9am EDT Wednesday. We are working to comply completely with these new requests with these vendors and are bringing in additional cleaning personnel onsite to make the fire marshal's deadline.
In preparation for being able to allow clients onsite, the fire marshal has stated that we need to perform a full test of the fire/life safety systems which will be done after utility power has been restored and fire system components replaced. We have these vendors standing by for this work tomorrow.
Assuming that all goes as planned, the earliest that clients will be allowed back into the site to power up their servers would be late in the day Wednesday.
TL;DR, evening Wednesday then specific timing will vary by provider after that. Good news, it sounds to me like the fire did not spread to customer equipment, and fingers-crossed for everyone's equipment for any damage from power spike, abrupt power loss, etc.
Hopefully these "additional cleaning personnel" & "vendors standing by for this work tomorrow" are fully approved by the marshal
Ooooff not great news.
I feel the blame being put on the fire marshall is kind of like the whole Covid excuse being over played.
I could be completely wrong, but at the end of the day the marshall is going to only allow things to come online when it's safe to actually do so. That's the only thing they care about and the fact that we hear smoke was seen and then a fire and can't remember now if the staff on site put it out or someone else. But sorry if you see smoke in the UPS room shit's pretty bad already at that point power should have been killed to that room.
Sadly in this industry and many others people will say anything to try and avoid taking the full blame when SHTF. I am interested to hear from anyone on site if they have seen more things or not.
I'm hoping for all of the companies who's customers who are down that this will get back online quickly for them, as this can affect some of the smaller companies even larger companies lives if their customers cancel due to a massive outage. We all know how some customers can be.
Mentally strong VPS provider immediately leases another hypervisor in another nearby datacenter and migrates affected customers without data.
Burned datacenter is the only one hit with cancellations, as they deserve.
I have some affected VPS servers with you. I feel for you guys and thanks for the updates ..
Ya I keep getting updated "Another inspection at 9am EDT Wednesday"
Sorry about your new email server
not new but ok
I see lol I think I need to change the group that its in. but I have had my data there for about a year now. so yes a Year new.
Same here.
The way things are progressing, I don't see it opening even today.
Hi we are in Brazil and we are served by Kamatera with servers in the same location. For us, a situation like this is unacceptable. I mean, apparently and from what you're describing there was a totally isolated incident and the Fire Department turned off the power to the whole building. They asked those responsible for repair and cleaning, returned for the next inspection and then requested NEW demands. I mean, why not order everything at once? Do they not understand that they are causing hundreds of thousands of dollars in loss to us and our customers? Do you have an estimate of how many sites or users are being affected?