New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Psychz Virginia down
Psychz in Virginia looks to be down hard - http://lg.va.psychz.net/ not responding
Comments
Apparently it's lightning in the area... so the datacenter decided to just not take any chances and pull the plug on the whole datacenter.
Hope lightning isn't very common in the area.
That is... an interesting approach? Lmao.
Just relaying what I was told...
Now, if you'll excuse me, I've got to go offline and disconnect everything... because the sun's shining here and it might melt.
Trump flushed wrong toiled. China closed line.
Boom. Down.
This is George from Psychz Networks. We've gotten word from the facility (atlantic metro IAD4) that during a storm, lightning struct their condensers and out of abundance of caution, power had to be cut to equipment to protect it from overheating.
We've not gotten an ETA on when this will be resolved, but be assured that we're in constant contact with the facility about this issues.
If you have any other questions, please open a ticket in our dashboard or DM me directly, ill do my best to assist and keep you updated.
We have been told -
We've gotten word from the facility (atlantic metro IAD4) that during a storm, lightning struct their condensers and out of abundance of caution, power had to be cut to equipment to protect it from overheating.
We've not gotten an ETA on when this will be resolved, but be assured that we're in constant contact with the facility about this issues.
You're supposed to have generators for events like this
Just to be clear.. I'm not involved with the datacenter. We just have a server in that facility and was relaying what I had been told.
What @GeoBeo said makes a little bit more sense, but that includes a key piece of information that we did not initially get (that the condenser had been hit). What I was told just made it sound like "EGADS!!! LIGHTNING... Quick... shut everything down!"
NVM was going to educate you, but if you can't be bothered to google then you have the problem not them.
That wouldn't help anything. If the condensers took a lightning hit, that means facility cooling is down, and even if your critical load has power via UPS/Generators, the facility would heat up fast, and gear would start getting fried. If this was serious enough to take out cooling like that, they are making the right decision to shut down the critical load. Design of cooling layout is in question, and lightning protection infrastructure, but that has nothing to do with current decision in light of the situation. Keep in mind this is on the facility owner to remedy, not necessarily Psychz. This was an act of god issue, and unfortunately shite happens. We are affected as well....hope resolved soon, but these situations usually are not a quick resolution.
.
It's shit that it's happened but shit happens.
Good to see it's not someone over reacting and it's actually something serious (cooling is very serious).
It's times like this I do like Anycast though
Trying to find some updates on this is proving difficult. I hope its hours, not days?
Word is 3-4 hours.
Yea, it's a bit concerning. But I can't imagine there being a lot of information to report. But would be nice to know if we are talking a couple more hours, 6 hours, 12 hours, days?
I'm assuming this means 3 to 4 MORE hours, since it's approaching 5 hours of being down for us.
Based on the explanation, if you have services there, I would for sure fall back to your back-up plan for now. Because of the large amount of expensive equipment at stake, until they can confirm the cooling systems are working as expected, they will not turn the equipment on (insurance and the hell that is replacing hardware in thousands of machines that fried hardware because you let them run with no cooling).
Additionally, once this is rectified, then usually they turn hardware back on in stages, so this means even once they have the go ahead, it will take some hours to bring all equipment back online and restore all services.
my 2 cents.
Cheers!
@sparek yes I doubt the facility has time travel
Patience is a virtue.
This is after cooling issues on 05/11/19, 05/12/19, and 08/12/19. On the first two, cold side air got up to close to 120F. We didn't detect temp rise on 08/12/19, but they proactively notified us.
This time they sent out an email that we have 5 minutes to shut down our servers, but that email was received 3 minutes after they shut down power.
That facility has gone way downhill since InfoRelay was purchased by Atlantic Metro.
We will be looking for a new provider next week.
You still received an email from them, we have received nothing until we opened a ticket. Still waiting for their updates now.
Servers be getting some electrons
They were just testing overclocked servers.
Where is the N+1?
looks like N = 0 in this case. This and similar drama makes let so great to read
We received the below update: We have installed spot coolers around the datacenter to facilitate cooling and are in the process of installing duct work to accommodate a larger mobile cooling unit. Once this larger unit is running we will be able to bring the temperature down in the data center to a safe operating levels. Once the room temperatures have stabilized we will begin power on our floor PDUs in a controlled manner so as to monitor temperatures as we add heat to the room.
It looks to me like days and probably weeks before any services are restored.
Their looking glass was online around 2 hours ago so I hope that they can restore their services soon. ~ 11 hours of downtime already.
We are balancing the cooling temperatures within the Data Center so equipment does not overheat and cause any damage to the powered on equipment. This will continue for the foreseeable future.
Looks like its a "Disaster center" with no "foreseeable future"
Maybe their is N +1 but the air gap between the condensers was not big enough, still hit booth of these.
Its bad, if a lightning even hits these, they have lightning rods or?
I was told that they (Atlantic Metro) can't accept new orders in IAD4 so I chose their IAD3 (actually CoreSite VA1) location instead and we aren't affected at all in this case.
:-D
Was lightning the issue when psychz LA went out three times in one week too?
Who said lightning never strikes thrice in the same place?