New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
I am sure ppl on site doing all they can to bring everything back asap.
Any Idea When The Facility Goes Live Again My Website And Software Are Down From Last 48 Hours
This for a few hours to days of downtime?
And no redundancy in place?
That's really stupid
Are the back to cleaning updates again? I haven't checked the status for a while now.
Would be nice to have somebody to setup live stream from some building nearby to check whats going on realtime.
I'm interested into what providers are affected by this.
Most I've read so far has been:
If this is the place https://youtube.com/watch?v=bm456UOo2-s then nothing will stop if theres a Fire! No backup will help in the video they show off a Nice setup but when it comes to a Fire in One room! well nothing is going to help. I am happy its about over. this post has more of what's going on then I am being told. I thank all who keep us updated the best you can. yes it be nice to have some feet on the ground AKA some News corp or Youtuber.
@moodwriter last update:
https://status.dedicated.com/incidents/44
we have 11 servers with more then 3500 shared hosting clients down from last 48 hours due to these guys.
Video description says: 2 Emerson Lane, Secaucus, NJ. Which I believe is a different.
EWR1 is located at 1 Enterprise Avenue N, Secaucus, NJ
They appear to be about 0.5 miles away from each other though.
two hours still no update with the hourly updates
Im trying to be optimistic and learned my lesson here regarding redundancy, it was a very very very hard pill
Guys follow latest update of
@ethernetservers. I don't know what time zone they are in but they are 4 hours ahead of me, I believe 5hrs compared to NY:
he latest update from the data center:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting us with this. We have brought UPS-4 online and are currently charging it's associated battery system. While bringing UPS-R online we have run into a minor issue that we are currently investigating.
We are currently at 35% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 5 hours. We are still planning for an evening time frame when clients will be able to come back on site.
In process with our power system re-energizing, we have been working on our fire system as well. We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We are currently sourcing additional personnel to assist us with this escort policy.
@toast555
Thanks saw that too. I am a direct customer of evocative and im trying to tell myself itll be back but I dont know. I keep telling myself ill go on site in 3 hours but that 3 hours keeps getting pushed.
I hope they are being genuine saying they are still planning to let people in today.
I also hope the power is restored to all racks and not just some.
Yes, that's what we expect too. We are Kamatera customers and they are not reporting any updates. Which is very strange to me.
A key factor in all of this is just let your customers know even if it's nothing new that you know. It's super stressful when there are not updates coming out the best thing is to have hourly updates! It's a little weird that they haven't gotten everything back online if they got the ok from the fire team.
If things are not back online in a few more hours I as suspected there was a lot more to the story than that being told to everyone.
I would love to hear more from anyone local who has tried to go on site.
I to am with Kamatera and I have been getting email updates you should open a Ticket to keep you Updated That what I did. but as for Kamatera if you don't ask dont tell kinda thing? haha IDK
"Dear valued customers, We are pleased to inform you that we have received the latest update from the data center in New York, indicating that the electricity supply has been restored and tests are currently underway. Once we receive confirmation of normalcy, the cooling process will commence, which is estimated to take approximately 5 hours. Following the completion of the cooling process, the servers will be brought online. We sincerely appreciate your patience throughout this period and would like to thank you once again."
Kind Regards,
Daniel Chupin
System Engineer
Kamatera Inc.
@basskillin Yes. I already opened a ticket. As soon as the procedure started I received the information that it would take 4 to 5 hours and nothing else. It's ridiculous but when asked again for updates the answer was that there were no updates and so I have to prove it by showing a print of the Ethernetserver customer area or the link with the status of dedicated.com. If they update why is Kamatera no longer active for us who are their customers? Looks like negligence to me.
@PureVoltage Yes I agree. It would be very interesting if someone who is here in the comments could go to the place and tell us what they saw there.
I run uptime-kuma I know something was up jest did not know what until I logged to kamatera and seen "Server operation currently disabled, please try again soon." then I asked whats going on and it took them no time to reply. I do think kamatera has there Servers there like others.
@basskillin
I really don't know what their problem is with neglecting information this way. Whether it is a company protection policy or prevention of a future problem that makes reconnection unfeasible. But check below the last update made 8 minutes ago by dedicated.com
DC Update:
I plan to be there in about 2-3 hours
I agree. I would love to see this famous UPS room that supposedly needed all of the cleaning.
Nice @toast555 It would be amazing. And if you can tell us what to see there, that would be great. This whole situation, in addition to being stressful, generates a breach of trust. The installations are fantastic but I'm leaving this DC.
Please do update us on how it goes!
From what the datacenter is saying, you may need to put yourself on a list to be escorted in.
Do you know how the powerup process will go? Will they provide power to each tenant upon request? And what happens to the tenants who don't have somebody on the ground? Will the DC staff power up their cabinets/racks without being asked?
Interesting question @aj_potc I will submit it on my Kamatera ticket. Despite knowing what the answer will be: "we don't know."
I dont know the exact way to get put on a list.
I also have no idea about the power up aside from what they posted on status.dedicated.com they are relaying the exact info evocative is emailing and telling customers
In one of the email they said they will advise how to get remote hands support for customers who dont have somebody there.
Part of the problem is some companies don't want to post or don't post what they don't know as I am sure they're being fed the same bs over and over or not getting updates quickly either.
This is why it's still good to have a page or twitter etc that you post hourly updates even if it's "Still waiting on the next update"
Those things help prevent customers complaining. It could also be that at this time we've seen 3-4 hours guys, 3-4 hours etc and it's still all down. At the end of the day everyone who is down just wants to know it's back up and working.
Olá! From Portugal
Incidents like this one may happen to any DC. You shouldn't leave them just because this happened. The fire marshal has to be sure they comply to the safety rules, but again, this may happen to any other DC l.