New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Fire in electrical room of Dedicated.com in New York
We've been informed that an electrical room experienced a fire, was put out by retardant, and the datacenter is in emergency power off status at the requirement of on-site fire fighters. Our COO is on site, but we are presently unable to access the building. We are working to learn more. We do not have further information at this time. As this outage does not have a clearly defined resolution and is outside of our control at this time, please execute your business continuity plans accordingly. We will continue to post updates here as we have them.
Happened a couple of hours ago and seems to be under control now. All VirMach's NY nodes are down.
Comments
Why in aprox 2 years , start fire lot of data centers?I noticed that most of them come from UPSs and batteries, recently these data centers have updated their batteries and the manufacturer has several faulty batches?
So this is just a fire issue in the INAP/Evocative DC in general (specific to electrical room, seems to be put out, but will likely have extended downtime for repairs)?
Very unfortunate but good that it's isolated and hopefully didn't harm any customer hardware.
Could be anything really. Summers have overall been a lot hotter than usual.
Also likely that some hosts maybe cost cutting on maintenance trying to keep the lowest prices in these times of inflation.
We all know the latter was the case at OVH.
as earth temperature has risen by 33.F (1 degree C) in last decade is set to rise by 2 to 3 degree celsius (37.4 F) in coming 5-6 years we will see such fires in buildings (where electrical equipment are used) quite often henceforth.
DC's should invest more in keeping the servers/equipment and overall internal temperatures more cooler.
This corresponds to my NY HostHatch outage. Dashboard now says:
My server went offline at 2023-07-10 17:34:36 CT.
There was a brief network only outage 2023-07-10 16:50:19 -> 17:01:07.
Yup. DediPath also uses the same datacenter.
First, 1 degree of Celsius is not 33 degrees of Fahrenheit. 1 degree of C is 1.8 degrees of F, so 2-3 celsius is 3.6 - 5.4 F.
1C=33F, but the world isn't heading up by 37.4F over the next 5-6 years. That'd be an increase of 20.7C. The sidewalks would melt.
Anyway, I'm skeptical that global warming is really going to have such an effect on DCs. These rooms cool by 40-50 degrees F already. Another few degrees will push them over the edge?
This is impacting https://www.ethernetservers.com as well. They advised that the next update will be at 8 AM EDT.
Isn't NY under the heavy rain and floods? Should be easy to put out that fire...
And long time I have impression that dedicated.com is premium.
Wait, virmache nodes were up ?
Yeah, definitely agree. A few degrees outdoor ambient won't make enough of a difference for the DC's. They will just shed load appropriately or add cooling if it really mattered. Almost certainly more likely a combination of factors along the lines of:
-Aging equipment and facilities: a lot of DC's had their initial builds/rounds of investments >10 years ago. Stuff is just starting to age out (some of which might not be readily PM'd items) and finding new and creative ways to fail.
-A weird mix of a lot of talent retiring/being laid off while simultaneously not being able to be replaced with qualified candidates (or nobody is there to properly train the new people). Could be a number of reasons, but on small scale I've seen VC/public money come in with their c-suite of choice that will increase the bottom line but generally seems to be somewhat shortsighted for operational stuff. Or they've calculated and figured the risk of the outage/failure is just worth the savings y/y as it's pretty hard for their customers to migrate racks full of live servers.
-A lot of parts and hardware from the 'covid days' had piss poor QC/checks/controls. Chalk up any one of the dozen reasons you can legitimately attribute to this, but it wouldn't surprise me if we see a bit more of this as those faulty parts and workmanship find their way out of the ecosystem.
Finally, there are just multiples more datacenters/servers live at any given moment today than 10+ years ago. I wouldn't be surprised if the per-capita (per kW, per device, whatever) incidents are lower today than >10 years ago. I suppose we also are just so heavily reliant on connectivity for everything in your day-to-day life that you notice the impact of an outage that much more.
Note to be taken: the best DC's are taken/built by the internet moguls: AWS, MS and others.
You forgot about me
not realy. best would be something like iron mountain
The problem is mostly the water reserves drying up in many places and ocean rising by few meters.
Then there's the more commonly seen heat waves. Nothing really that humanity could not withstand with some infra costs, but its really something that can't be avoided until we are able to oil/gas/coal as energy source.
The more we pollute in this decade, the more green energy is needed in later on to offset the changes we cause.
We need more nuclear energy, but before that, we need more final disposal places like Finland's Onkalo for the nuclear waste.
That's some al gore level hoax conspiracy. The only damaging pollution is the politics.
I wish anyone affected by this fire all the love in the world
Can confirm, AWS and Azure have issues too. Didn't Azure's UK DC overheat and need to be taken down for a few hours?
This DC is really used by a lot of companies, there are many more not mentioned here.
Lord Eddard Stark be commemorated for fighting the perils of Winter. For Summer is Coming!
HBO should definitely release a sequel to the series, to take into account of the very real, double-proven, fact-checked facts about ClimateChange (with camel case). When in doubt folks, trust the science™, and may God help us from the heavens thru the sacrifice of a young virgin.
...Volunteers?
But in all seriousness, I hope they manage to contain the datacenter fire and I'm sure you all have multiple backups of your most important data, just like you always do. Right?
Lord Eddard Stark in his royal beachwear: "Summer is coming!"
Just for fun don't take this seriously!
Yep. I've got a dedicated server down and I'm trying to decide what I want to do. I was too busy last night to deal with it. Fortunately, what's on there is not super critical yet, otherwise I would have implemented my emergency plan by now. So I decided to wait overnight and wanted to see how this would play out. But at this point it's taking way longer than I expected.
I now wonder if I should be shopping for somewhere else. I had another downtime there about three weeks ago, nothing to do with my server, right in the middle of business, so the data center is not inspiring confidence. My server itself has been reliable, but the data center has not been great for me in recent months.
Anyone else considering jumping ship from this data center? Or do you think things will be better now that they had such a significant outage and they'll be under more scrutiny?
yes.
people think google, aws and microsoft datacenters are ALL tier 3 or 4 like but in reality some of them were not even tier 1.
if you put the profits of aws, google or microsoft in relation to their data centers, i.e. how some of them were built or are still being built and what kind of incidents there were, you quickly realize that many things are not so "premium" after all.
"but they are all DoD certified" yes because exactly these three sell any data they somehow get to the us government or better said various us authorities. they shit on any data protection laws that are not given by the us government and break them again and again because the penalties for this are laughable.
Looks like the whole Evocative New Jersey Data Center (EWR1) affected.
Dedipath, Ethernetservers, VirMach
My VPS's from these providers (NY location) offline, unreachable.
Details from DediPath: https://i.imgur.com/V25a2xA.png
yeah my ethernetserver vps and dedipath shared is down, such is luck
Other providers (or anyone else, this has been good exercise for our status page, which hasn't been exercised like this before): please feel free to subscribe to the issue directly at the bottom of the page: https://status.dedicated.com/incidents/44 - we seem to be getting more consistent updates from Evocative than others.
Yes, this is my first post in years. No, I'm not going to post any more.
My Hawkhost's and Ethernetserver's hosted sites are down (19+ hours and counting).
Same for me. This is by far the longest downtime I've had with Hawk Host since 2009.
Those, who look for another source of updates on New York Secaucus data center fire, can check https://hawkhoststatus.com/
This is the latest update from Dedipath.
Also got an update from RoyaleHosting from 1 hr ago:
Lol, have you watched the recently released prequel? It's mid at best, unfortunately.