Nocix issues? Servers not reachable?

Comments

• Advin Member, Host Rep

  June 1 edited June 1

  KCIX is partially down as well. Our cabinets at 1530 Swift St and network (which is via NOCIX) is online though.
• MannDude Patron Provider, Veteran

  June 1

  No issues in our space at 1530 Swift. I do see KCIX traffic dropped to nothing a couple hours ago though.
• EthernetServers Member, Patron Provider

  June 1 edited June 1

  According to a post on WHT, someone received this:

  I regret to inform you that we are experiencing major power issues. We are currently working to to resolve the issue and get you back up as soon as we can. Unfortunately we have no ETA but will let you know once the power issue has resolved.

  Thanked by 4oloke satorik fatchan brianrizzo
• orangevps Member, Patron Provider

  June 1

  Yes,

  We're affected. Mostly Asia

  Thanked by 1oloke
• OhJohn Member

  June 1

  one up, one down. So yes, seems partial outage.
• fatchan Member, Host Rep

  June 1 edited June 1

  Partial outage. Some subnets stopped being announced but came back in the past ~30 mins. Servers still not reachable.

  A local power company outage map shows storm damage in areas around their address.

  Thanked by 2satorik hostal
• jeanbar Member

  June 1

  Quick update:

  I received the following response from Nocix support about 2h ago:

  "We unfortunately suffered a major power failure recently and are working to get everyone back online. Please standby while we recover from this incident and we will get you back online as soon as possible."

  At this point, no ETA has been provided. I'm still waiting for further updates and will post any new information here as soon as I receive it.

  Is anyone else affected by this outage?

  I'll share additional updates if I receive more information.

  Thanked by 1fatchan
• fatchan Member, Host Rep

  June 1

  About 1/3 of my servers have come back online so far.
• jeanbar Member

  June 1

  Quick update:

  My server is back online after approximately 8 hours of downtime.

  I don't know if this information helps others identify the affected infrastructure, but my server location is:

  Server Location: S3HHU45

  Hopefully this can help other customers determine whether their servers are in the same affected area.

  Has anyone else recovered service yet?
• acidpuke Member

  June 1

  My server was down until now, I went on the control panel and tried a reboot about 5 mins later it came back online. About 8 hrs of downtime also.
• vinhais Member

  June 2

  My server was down for about 7 hours and 48 minutes, but since it was just a development environment, and I got double the resources when I signed up, I can't complain about anything.
• digunix Member

  June 2 edited June 2

  I still have a server down and support just keeps giving unhelpful responses.
  Server Location: S3FNU3
• digunix Member

  June 2

  We are back online finally after 36ish hours... felt like a lifetime.
• fatchan Member, Host Rep

  June 3

  Cross posting an update by Aaron from WHT:

  First off, let me apologize for not posting sooner. Honestly I'm not on WHT much any more and, with everything going on, didn't even think about it until now.

  Early Monday morning our facility was hit with, what the National Weather Service is calling a microburst. I've been in several microbursts in my life and I've never known them to toss AC units around and rip all the stop signs around the building out of the ground but I guess I'm not the expert. Standing in the facility at the time it sure seemed like the air was moving in a circular pattern around the buildings...

  In any event, three of our six outside power facilities were damaged. This caused 2 things to happen: We had a partial power outage to the DC floor and we lost AC units in another part. In the first instance, dedicated servers lost power. In the second, AC capacity was lost and servers were intentionally shut down to mitigate thermal runaway. This resulted in the powering down of about 10,000 servers. Power has been fully restored to all affected facilities

  The majority of our facility was unaffected. Undamaged backup systems operated as designed and the majority of our business stayed online and went unnoticed.

  Of the, roughly, 10,000 servers affected, we are still working through 319 (as of this writing). These are mainly power supplies and hard drives that didn't come back after they were powered off. Technicians are working over time and colocation techs have been reassigned to dedicated server support. I would expect the ticket queue to be mostly clear by EOB tomorrow.

  If you can ping your server but your website isn't up, please make sure your VM is started if you have one. I'm told we have a large volume of tickets where that was or is the issue.

  I thank you for your patience. I, honestly, can't remember the last time we had this many servers offline at once.

  If you are a colo cusotmer: Yes. Our colo facilities sustained damage, none of which was customer affecting.

  I should note that I haven't read this thread so if there were questions posed to me I haven't seen them.

  Dedicated server support is handled through our ticket system only. As of right now, there are no tickets that haven't been responded to.

  Colocation customers have our escalation matrix which includes my personal mobile number but the one of you that was affected got a call from me at 2am yesterday morning.

  Guess my last 2 servers fall in that 319 or so. Luckily not a big deal, and have backups in case of damaged drives.

  Thanked by 1MikeA