Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Psychz Virginia down
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Psychz Virginia down

Psychz in Virginia looks to be down hard - http://lg.va.psychz.net/ not responding

«1

Comments

  • Apparently it's lightning in the area... so the datacenter decided to just not take any chances and pull the plug on the whole datacenter.

    Hope lightning isn't very common in the area.

  • @sparek said:
    Apparently it's lightning in the area... so the datacenter decided to just not take any chances and pull the plug on the whole datacenter.

    Hope lightning isn't very common in the area.

    That is... an interesting approach? Lmao.

    Thanked by 1Ouji
  • Just relaying what I was told...

    Now, if you'll excuse me, I've got to go offline and disconnect everything... because the sun's shining here and it might melt.

  • Trump flushed wrong toiled. China closed line.
    Boom. Down.

  • GeoBeoGeoBeo Member, Patron Provider

    This is George from Psychz Networks. We've gotten word from the facility (atlantic metro IAD4) that during a storm, lightning struct their condensers and out of abundance of caution, power had to be cut to equipment to protect it from overheating.

    We've not gotten an ETA on when this will be resolved, but be assured that we're in constant contact with the facility about this issues.

    If you have any other questions, please open a ticket in our dashboard or DM me directly, ill do my best to assist and keep you updated.

  • AlexanderMAlexanderM Member, Top Host, Host Rep

    We have been told -

    We've gotten word from the facility (atlantic metro IAD4) that during a storm, lightning struct their condensers and out of abundance of caution, power had to be cut to equipment to protect it from overheating.

    We've not gotten an ETA on when this will be resolved, but be assured that we're in constant contact with the facility about this issues.

  • @sparek said:
    Apparently it's lightning in the area... so the datacenter decided to just not take any chances and pull the plug on the whole datacenter.

    Hope lightning isn't very common in the area.

    You're supposed to have generators for events like this

  • @terrorgen said:
    You're supposed to have generators for events like this

    Just to be clear.. I'm not involved with the datacenter. We just have a server in that facility and was relaying what I had been told.

    What @GeoBeo said makes a little bit more sense, but that includes a key piece of information that we did not initially get (that the condenser had been hit). What I was told just made it sound like "EGADS!!! LIGHTNING... Quick... shut everything down!"

  • @terrorgen said:

    @sparek said:
    Apparently it's lightning in the area... so the datacenter decided to just not take any chances and pull the plug on the whole datacenter.

    Hope lightning isn't very common in the area.

    You're supposed to have generators for events like this

    NVM was going to educate you, but if you can't be bothered to google then you have the problem not them.

  • SmartHostSmartHost Patron Provider, Veteran

    You're supposed to have generators for events like this

    That wouldn't help anything. If the condensers took a lightning hit, that means facility cooling is down, and even if your critical load has power via UPS/Generators, the facility would heat up fast, and gear would start getting fried. If this was serious enough to take out cooling like that, they are making the right decision to shut down the critical load. Design of cooling layout is in question, and lightning protection infrastructure, but that has nothing to do with current decision in light of the situation. Keep in mind this is on the facility owner to remedy, not necessarily Psychz. This was an act of god issue, and unfortunately shite happens. We are affected as well....hope resolved soon, but these situations usually are not a quick resolution.
    .

  • SplitIceSplitIce Member, Host Rep

    It's shit that it's happened but shit happens.

    Good to see it's not someone over reacting and it's actually something serious (cooling is very serious).

    It's times like this I do like Anycast though :)

  • SmartHost said: but these situations usually are not a quick resolution

    Trying to find some updates on this is proving difficult. I hope its hours, not days?

  • SplitIceSplitIce Member, Host Rep

    Word is 3-4 hours.

  • @epaslv said:
    Trying to find some updates on this is proving difficult. I hope its hours, not days?

    Yea, it's a bit concerning. But I can't imagine there being a lot of information to report. But would be nice to know if we are talking a couple more hours, 6 hours, 12 hours, days?

    @SplitIce said:
    Word is 3-4 hours.

    I'm assuming this means 3 to 4 MORE hours, since it's approaching 5 hours of being down for us.

  • epaslv said: Trying to find some updates on this is proving difficult. I hope its hours, not days?

    Based on the explanation, if you have services there, I would for sure fall back to your back-up plan for now. Because of the large amount of expensive equipment at stake, until they can confirm the cooling systems are working as expected, they will not turn the equipment on (insurance and the hell that is replacing hardware in thousands of machines that fried hardware because you let them run with no cooling).

    Additionally, once this is rectified, then usually they turn hardware back on in stages, so this means even once they have the go ahead, it will take some hours to bring all equipment back online and restore all services.

    my 2 cents.

    Cheers!

    Thanked by 1uptime
  • SplitIceSplitIce Member, Host Rep

    @sparek yes I doubt the facility has time travel

  • PieHasBeenEatenPieHasBeenEaten Member, Host Rep

    Patience is a virtue.

  • This is after cooling issues on 05/11/19, 05/12/19, and 08/12/19. On the first two, cold side air got up to close to 120F. We didn't detect temp rise on 08/12/19, but they proactively notified us.

    This time they sent out an email that we have 5 minutes to shut down our servers, but that email was received 3 minutes after they shut down power.

    That facility has gone way downhill since InfoRelay was purchased by Atlantic Metro.

    We will be looking for a new provider next week.

  • NDTNNDTN Member, Patron Provider, Top Host

    @joebg said:
    This is after cooling issues on 05/11/19, 05/12/19, and 08/12/19. On the first two, cold side air got up to close to 120F. We didn't detect temp rise on 08/12/19, but they proactively notified us.

    This time they sent out an email that we have 5 minutes to shut down our servers, but that email was received 3 minutes after they shut down power.

    That facility has gone way downhill since InfoRelay was purchased by Atlantic Metro.

    We will be looking for a new provider next week.

    You still received an email from them, we have received nothing until we opened a ticket. Still waiting for their updates now.

  • SplitIceSplitIce Member, Host Rep

    Servers be getting some electrons

  • MikeAMikeA Member, Patron Provider

    @SplitIce said:
    Servers be getting some electrons

    They were just testing overclocked servers.

  • ClouviderClouvider Member, Patron Provider

    Where is the N+1?

  • First-RootFirst-Root Member, Host Rep

    @Clouvider said:
    Where is the N+1?

    looks like N = 0 in this case. This and similar drama makes let so great to read :)

    Thanked by 1Clouvider
  • We received the below update: We have installed spot coolers around the datacenter to facilitate cooling and are in the process of installing duct work to accommodate a larger mobile cooling unit. Once this larger unit is running we will be able to bring the temperature down in the data center to a safe operating levels. Once the room temperatures have stabilized we will begin power on our floor PDUs in a controlled manner so as to monitor temperatures as we add heat to the room.

    It looks to me like days and probably weeks before any services are restored.

  • NDTNNDTN Member, Patron Provider, Top Host

    Their looking glass was online around 2 hours ago so I hope that they can restore their services soon. ~ 11 hours of downtime already.

    @epaslv said:
    We received the below update: We have installed spot coolers around the datacenter to facilitate cooling and are in the process of installing duct work to accommodate a larger mobile cooling unit. Once this larger unit is running we will be able to bring the temperature down in the data center to a safe operating levels. Once the room temperatures have stabilized we will begin power on our floor PDUs in a controlled manner so as to monitor temperatures as we add heat to the room.

    It looks to me like days and probably weeks before any services are restored.

  • We are balancing the cooling temperatures within the Data Center so equipment does not overheat and cause any damage to the powered on equipment. This will continue for the foreseeable future.

    Looks like its a "Disaster center" with no "foreseeable future"

  • NeoonNeoon Community Contributor, Veteran

    @Clouvider said:
    Where is the N+1?

    Maybe their is N +1 but the air gap between the condensers was not big enough, still hit booth of these.

    Its bad, if a lightning even hits these, they have lightning rods or?

  • avelineaveline Member, Patron Provider

    I was told that they (Atlantic Metro) can't accept new orders in IAD4 so I chose their IAD3 (actually CoreSite VA1) location instead and we aren't affected at all in this case.

    :-D

    Thanked by 1feezioxiii
  • Was lightning the issue when psychz LA went out three times in one week too?

  • @9gigsofram said:
    Was lightning the issue when psychz LA went out three times in one week too?

    Who said lightning never strikes thrice in the same place?

    Thanked by 1NobodyInteresting
Sign In or Register to comment.