NOCIX down

pitbullrv · July 2023

@ratatuilekkk666 said:
Update, they load System Rescue 9 for me, and with i'm recovering the files and makings backups..
In one server, another server still dead, and no answer from support people..

Friend, you said: "Yep, i tried another server, loading the system rescue, and now is ping works.." (This is my problem, server inacessible by ping or SSH)

Did you do this through the host panel? In mine, I have this option:

But I'm afraid that after running this, there might be a risk of permanently losing my files. What do you think?

ratatuilekkk666 · July 2023

@pitbullrv said:

@ratatuilekkk666 said:
Update, they load System Rescue 9 for me, and with i'm recovering the files and makings backups..
In one server, another server still dead, and no answer from support people..

Friend, you said: "Yep, i tried another server, loading the system rescue, and now is ping works.." (This is my problem, server inacessible by ping or SSH)

Did you do this through the host panel? In mine, I have this option:

But I'm afraid that after running this, there might be a risk of permanently losing my files. What do you think?

My server was not loading the OS because of a corruption into file system..
Just after booting into sysrcd that ping works..

booting into sysrcd will not cause file or HD deletion..

just if you select a other OS.. haha

sysrcd will be loaded into the memory, like a live CD..

pitbullrv · July 2023

It doesn't work, it doesn't load the system through the hosting panel, if someone is going to try, don't even try.

"Hello

Your OS reload has been scheduled. Please allow up to 45 minutes for the reload process to complete.

Thank You"

I've been waiting for two hours and it doesn't complete the task.

ratatuilekkk666 · July 2023

@pitbullrv said:
It doesn't work, it doesn't load the system through the hosting panel, if someone is going to try, don't even try.

"Hello

Your OS reload has been scheduled. Please allow up to 45 minutes for the reload process to complete.

Thank You"

I've been waiting for two hours and it doesn't complete the task.

yeah, i have one server that taking more than 45 minutes too, but eventually works..

i believe they automated system is not consistent and standardized.. they have different controllers for each racks, or whatever..

Some of my server with them works, some have sysrcd v4, some v9..

pitbullrv · July 2023

My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

ratatuilekkk666 · July 2023

@pitbullrv said:
My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

Yeah, same message here.. this Cole is an ass..

The other support guys are much more helpful

MrRadic · July 2023

@ratatuilekkk666 said:

@pitbullrv said:
My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

Yeah, same message here.. this Cole is an ass..

The other support guys are much more helpful

They're stressed, give them time.

Hivelocity · July 2023

@MrRadic said:

@ratatuilekkk666 said:

@pitbullrv said:
My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

Yeah, same message here.. this Cole is an ass..

The other support guys are much more helpful

They're stressed, give them time.

And the guy's answering the tickets are 1. likely saying exactly what they have been told to say, 2. likely have nothing to do with the what caused the issues in the first place and 3. working harder than anyone right now.

Hivelocity · July 2023

This was just posted on WHT from someone who appears to represent NOCIX or one of the responsible entities.

First off. I apologize for the recent outage. It was a set of circumstances that should never happen.

While we're still working on figuring out all the details the short story is that power to the facility was knocked out by storm damage in the area. When this happened, generator one, which had just been tested the day before, started, then threw an overcrank error and shut down. Generator two started and ran as expected for a short time before we believe it was struck by lightning, damaging the alternator. This caused the phase to ground voltages to crazy, throwing alarms on the UPS' which then dropped load causing a complete outage to our entire facility.

This is the first complete outage we've experienced in our "new" facility (it's over 10 years old now). The last one I can think of was in 2008 or maybe 2010 when we were still downtown. That's not to say its the first time we've lost power. We lose power every once in awhile, like anywhere, and the generators run, UPS's work and no one ever knows.

So the issue becomes, when you have over 15,000 servers running, sometimes for years, along with PDUs, switches, etc, sometimes running for over 10 years, that suddenly get shut off they don't necessarily come back when the power comes back. Power supplies fail, drives fail, PDUs die, all sorts of bad things happen and it's not as easy as just getting the lights back on.

All the downed switches and PDUs were replaced by mid day yesterday. We're down to just the individual machines that have hardware issues that need to be fixed. When I say "just" I mean we're down to under 500 machines still offline.

I understand that getting responses to tickets that seem canned or short can be frustrating but every one wants a response and if we're spending all our time writing individual responses then we're not getting things turned back on. In addition, the technician answering your ticket may not have all the facts and we don't want them assuming things and giving out incorrect information. That leads to more confusion and just fosters distrust in ALL the information being disseminated.

As for excuses, I don't have any. What happened happened and we will look into way to improve as we always do. If you are still down I would ask that you work with our people in the original ticket you submitted. Submitting multiple new tickets just slows down the whole system for everyone and will actually increase your resolution time, not shorten it. Also, if you are not a direct customer please do not submit a ticket to our system. Work through your reseller. We don't have any information on you and therefor can't answer your questions.

Again, all overtime for our technical staff is approved and they are all here working as fast as they can. We expect to be down to a handful of servers by the end of the day and back to normal for the weekend.

ratatuilekkk666 · July 2023

@Hivelocity said:

@MrRadic said:

@ratatuilekkk666 said:

@pitbullrv said:
My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

Yeah, same message here.. this Cole is an ass..

The other support guys are much more helpful

They're stressed, give them time.

And the guy's answering the tickets are 1. likely saying exactly what they have been told to say, 2. likely have nothing to do with the what caused the issues in the first place and 3. working harder than anyone right now.

But you know what the major problem is?
Yesterday i ask for then to put available a System Rescue 10, which is the last version, or 9 work just fine..
They don't have a good automated system and consistency between the servers( Basically a mess, probably a bunch of older servers without any label and total chaos)..

The most of their clients i believe, have some knowledge with how to handle a few's commands line..

i'm on hold because of that.. in another server with then i asked via support ticket to load the system rescue 10, and now i'm backing up all the files to another server.

i will note this on your ticket subject for the server to be scheduled for a reload to Sysrcd 9 (we're still working on getting 10 setup on our automated system)
They say that yesterday.. but nothing yet, and the pressure is building up, because not even the clients can try to fix the problem

ratatuilekkk666 · July 2023

@MrRadic said:

@ratatuilekkk666 said:

@pitbullrv said:
My only progress is getting copy-pasted messages like this from several different agents.

But they are answering me. I think this is a positive point in the midst of chaos. But with no apparent solution.

Yeah, same message here.. this Cole is an ass..

The other support guys are much more helpful

They're stressed, give them time.

Yeah, its ok, but is a frustration that your problem is so simple that you can do it yourself..

But they have a ancient system with system rescue version from 2014 that don't work anymore with current versions of file system..

just a simple fsck, or xfs_repair..

pitbullrv · July 2023

We need wait bro, our hands are tied. If it doesn't resolve by the weekend, I'll consider my files lost and move on... In fact, I'm already in the recovery process using cloudflare's cache to save what can be saved.

ratatuilekkk666 · July 2023

@Hivelocity said:
This was just posted on WHT from someone who appears to represent NOCIX or one of the responsible entities.

First off. I apologize for the recent outage. It was a set of circumstances that should never happen.

While we're still working on figuring out all the details the short story is that power to the facility was knocked out by storm damage in the area. When this happened, generator one, which had just been tested the day before, started, then threw an overcrank error and shut down. Generator two started and ran as expected for a short time before we believe it was struck by lightning, damaging the alternator. This caused the phase to ground voltages to crazy, throwing alarms on the UPS' which then dropped load causing a complete outage to our entire facility.

This is the first complete outage we've experienced in our "new" facility (it's over 10 years old now). The last one I can think of was in 2008 or maybe 2010 when we were still downtown. That's not to say its the first time we've lost power. We lose power every once in awhile, like anywhere, and the generators run, UPS's work and no one ever knows.

So the issue becomes, when you have over 15,000 servers running, sometimes for years, along with PDUs, switches, etc, sometimes running for over 10 years, that suddenly get shut off they don't necessarily come back when the power comes back. Power supplies fail, drives fail, PDUs die, all sorts of bad things happen and it's not as easy as just getting the lights back on.

All the downed switches and PDUs were replaced by mid day yesterday. We're down to just the individual machines that have hardware issues that need to be fixed. When I say "just" I mean we're down to under 500 machines still offline.

I understand that getting responses to tickets that seem canned or short can be frustrating but every one wants a response and if we're spending all our time writing individual responses then we're not getting things turned back on. In addition, the technician answering your ticket may not have all the facts and we don't want them assuming things and giving out incorrect information. That leads to more confusion and just fosters distrust in ALL the information being disseminated.

As for excuses, I don't have any. What happened happened and we will look into way to improve as we always do. If you are still down I would ask that you work with our people in the original ticket you submitted. Submitting multiple new tickets just slows down the whole system for everyone and will actually increase your resolution time, not shorten it. Also, if you are not a direct customer please do not submit a ticket to our system. Work through your reseller. We don't have any information on you and therefor can't answer your questions.

Again, all overtime for our technical staff is approved and they are all here working as fast as they can. We expect to be down to a handful of servers by the end of the day and back to normal for the weekend.

lol, a generator failing and another struck by a lightning..
Crazy hahaha if is that true

ratatuilekkk666 · July 2023

@pitbullrv said:
We need wait bro, our hands are tied. If it doesn't resolve by the weekend, I'll consider my files lost and move on... In fact, I'm already in the recovery process using cloudflare's cache to save what can be saved.

Yeah, i started the damage control too..

deadpool · July 2023

(KC Native not speaking on behalf of Nocix) We had some crazy storm move through early morning July 5th. It was roughly 2 1/2 hours of 60mph winds and rain. Much of KC's suburbs have underground lines but downtown and within city limits use a lot of dated infrastructure. Surprised their generators didn't kick in and they're still working to restore the servers.

Evergy who I'm sure provides power (because they do to 99% of the metro) isn't showing any lasting outages and all current outages are small. https://outagemap.evergy.com/

ratatuilekkk666 · July 2023

Hello guys, just updating..
no updates at my problem with Nocix, they just need to boot in system rescue for me.. but...
A simple action, 3 days in doing..or more...

windytime90 · July 2023

@ratatuilekkk666 said:
Hello guys, just updating..
no updates at my problem with Nocix, they just need to boot in system rescue for me.. but...
A simple action, 3 days in doing..or more...

The same for me. Don't know what they are doing.

ratatuilekkk666 · July 2023

Well, finaly, they load the system rescue 9 for me, i was able to recovery all my files..
and then tried to do a fsck and reboot the system, but the corruption was real..

And then i just Reload Operating System, and now its all fine, starting from zero setting up the servers.

3 days for a simple boot into rescue mode..

jtk · July 2023

I have and have had a few systems with nocix for awhile. Generally things have been fine in my experience. This was a rare big, extended outage in my experience. I had one system that didn't come back right away from this event. They were responsive with me and finished "fixing" it within the past 24 hours. I had kicked off an OS reload, which would normally be done within an hour took about 24, and they made sure that happened.

You get best effort service, especially with the cheap budget gear here so I don't see anything out of the ordinary for I'd expect. A few days outage and no IP address changes at least, which is a lot better than I can say for some other providers I've seen who didn't even confront weather-related events.

pitbullrv · July 2023

@jtk said:
I have and have had a few systems with nocix for awhile. Generally things have been fine in my experience. This was a rare big, extended outage in my experience. I had one system that didn't come back right away from this event. They were responsive with me and finished "fixing" it within the past 24 hours. I had kicked off an OS reload, which would normally be done within an hour took about 24, and they made sure that happened.

You get best effort service, especially with the cheap budget gear here so I don't see anything out of the ordinary for I'd expect. A few days outage and no IP address changes at least, which is a lot better than I can say for some other providers I've seen who didn't even confront weather-related events.

Speak for yourself, I've been down for 5 days without any news from support.

ratatuilekkk666 · July 2023

@jtk said:
I have and have had a few systems with nocix for awhile. Generally things have been fine in my experience. This was a rare big, extended outage in my experience. I had one system that didn't come back right away from this event. They were responsive with me and finished "fixing" it within the past 24 hours. I had kicked off an OS reload, which would normally be done within an hour took about 24, and they made sure that happened.

You get best effort service, especially with the cheap budget gear here so I don't see anything out of the ordinary for I'd expect. A few days outage and no IP address changes at least, which is a lot better than I can say for some other providers I've seen who didn't even confront weather-related events.

inthecloudblog · July 2023

@Hivelocity steve , is that quote (when you refer about someone that represents nocix )from the guy that said :

“Several years ago I ran the numbers on what it would cost to replace my support staff with monkeys. The licenses, food, care, etc of the monkeys made each one cost about 4 times what I paid an average tech at the time so I abandoned the idea”
?

https://archive.is/7ZJL0
Edit:
I’m referring to Aaron

mhn · July 2023

I guess everyone's best bet that is with nocix is to move to @MrRadic and request same pricing in his latest "find a lower price and we beat it" thread

SirFoxy · July 2023

@mhn said:
I guess everyone's best bet that is with nocix is to move to @MrRadic and request same pricing in his latest "find a lower price and we beat it" thread

He doesn’t have a KC location. Think that only applies if it’s at the same location, because that’s a significant cost factor.

PureVoltage · July 2023

@vipeweb said:

@RFord said:
I think this is the longest downtime I've had since a certain hosting company loaded all the servers onto trucks and drove them to a new site in Nevada or somewhere.

It's a very cheap dedi but I'm going to have to cancel. Unfortunately I paid well in advance for another month which hasn't started yet.

I have 5 servers at nocix, I'm going to cancel them all

it was the most downtime I've ever had, almost 11 hours of downtime.

nocix forgets to pay for a few minutes, has to pay an additional $5, with all this inactivity how are we?

not even the ETA page felt like doing it

the impression I have is that they don't care, they think we have competitive prices, that's the price you pay.

i got 8 clients canceled the server with me

some joked asking if the server was hosted in my bedroom

The problem is people have come to expect 100% uptime from all providers.
Budget and ultra budget providers are just that. If you have clients hosted on something expecting no downtime and quick replies for not much money this is not the proper model for a business.

I believe a lot of their customers use them due to the cheap price and know they might have some large outages lasting longer than some other providers.

We've had customers open tickets due to a 60 second outage when routes switch providers due to an outage. Can't please everyone sadly.

That being said outages of any time suck. However, things do come up sadly in this industry.

ratatuilekkk666 · July 2023

@PureVoltage said:

@vipeweb said:

@RFord said:
I think this is the longest downtime I've had since a certain hosting company loaded all the servers onto trucks and drove them to a new site in Nevada or somewhere.

It's a very cheap dedi but I'm going to have to cancel. Unfortunately I paid well in advance for another month which hasn't started yet.

I have 5 servers at nocix, I'm going to cancel them all

it was the most downtime I've ever had, almost 11 hours of downtime.

nocix forgets to pay for a few minutes, has to pay an additional $5, with all this inactivity how are we?

not even the ETA page felt like doing it

the impression I have is that they don't care, they think we have competitive prices, that's the price you pay.

i got 8 clients canceled the server with me

some joked asking if the server was hosted in my bedroom

The problem is people have come to expect 100% uptime from all providers.
Budget and ultra budget providers are just that. If you have clients hosted on something expecting no downtime and quick replies for not much money this is not the proper model for a business.

I believe a lot of their customers use them due to the cheap price and know they might have some large outages lasting longer than some other providers.

We've had customers open tickets due to a 60 second outage when routes switch providers due to an outage. Can't please everyone sadly.

That being said outages of any time suck. However, things do come up sadly in this industry.

from my experience, some downtime is not the problem..
The problem with me, was the outdated automated system they have.. on my server they have Sysrcd 4.3.1 [rescue system].. that version is from 2014..
is that hard to keep a ISO file updated in their automated system?

Two of my servers get their file system corrupted, when they finally load the Sysrcd 9, i was able to mount the drives, backup all the files, and load into another server.. but only yesterday they load the Sysrcd 9, and that was a hard battle with the support guys...

i'm with nocix since datashack, and this was the first serious problem with them

SnowFields · July 2023

anyone else having power related issues? ever since this massive outage my servers keep losing power.

I struggle to get 2days uptime without it powering off

acefxlabs · July 2023

Is it common for Nocix to have downtime
I am just starting to use thier services

RFord · July 2023

I don't expect no downtime. But if hosting websites it starts to become worrying if longer than an hour. In my experience of using many hosts over the years, and low end ones for several years, it rarely lasts that long.

After about 13 hours down I cancelled and have no complaints about how NoCIX dealt with that. I was a bit sad about it as it was the first dedi I'd had.

@PureVoltage said:

@vipeweb said:

@RFord said:
I think this is the longest downtime I've had since a certain hosting company loaded all the servers onto trucks and drove them to a new site in Nevada or somewhere.

It's a very cheap dedi but I'm going to have to cancel. Unfortunately I paid well in advance for another month which hasn't started yet.

I have 5 servers at nocix, I'm going to cancel them all

it was the most downtime I've ever had, almost 11 hours of downtime.

nocix forgets to pay for a few minutes, has to pay an additional $5, with all this inactivity how are we?

not even the ETA page felt like doing it

the impression I have is that they don't care, they think we have competitive prices, that's the price you pay.

i got 8 clients canceled the server with me

some joked asking if the server was hosted in my bedroom

The problem is people have come to expect 100% uptime from all providers.
Budget and ultra budget providers are just that. If you have clients hosted on something expecting no downtime and quick replies for not much money this is not the proper model for a business.

I believe a lot of their customers use them due to the cheap price and know they might have some large outages lasting longer than some other providers.

We've had customers open tickets due to a 60 second outage when routes switch providers due to an outage. Can't please everyone sadly.

That being said outages of any time suck. However, things do come up sadly in this industry.

@PureVoltage said:

@vipeweb said:

@RFord said:
I think this is the longest downtime I've had since a certain hosting company loaded all the servers onto trucks and drove them to a new site in Nevada or somewhere.

It's a very cheap dedi but I'm going to have to cancel. Unfortunately I paid well in advance for another month which hasn't started yet.

I have 5 servers at nocix, I'm going to cancel them all

it was the most downtime I've ever had, almost 11 hours of downtime.

nocix forgets to pay for a few minutes, has to pay an additional $5, with all this inactivity how are we?

not even the ETA page felt like doing it

the impression I have is that they don't care, they think we have competitive prices, that's the price you pay.

i got 8 clients canceled the server with me

some joked asking if the server was hosted in my bedroom

The problem is people have come to expect 100% uptime from all providers.
Budget and ultra budget providers are just that. If you have clients hosted on something expecting no downtime and quick replies for not much money this is not the proper model for a business.

I believe a lot of their customers use them due to the cheap price and know they might have some large outages lasting longer than some other providers.

We've had customers open tickets due to a 60 second outage when routes switch providers due to an outage. Can't please everyone sadly.

That being said outages of any time suck. However, things do come up sadly in this industry.

emgh · July 2023

DataShack was my first dedi

Time sure flies🦋🦋

Howdy, Stranger!

Categories

In this Discussion

NOCIX down

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

NOCIX down

Comments