Incognet Kansas City Outage and WHMCS 403

Fourplex · 1:28PM

While Incognet is technically our rival, we do host a critical service there for redundancy (our WHMCS).

We have a Kansas City Incognet server and right now it's offline. Meaning we can't ping it or see it in Virtualizor. We have the Legay EPYC VPS.

And yet I cannot send a ticket to Incognet. I just get a 403 Forbidden when I try to send in one.

Could @MannDude or someone else check into this?

MannDude · 1:32PM

Portal isn't down, are you still unable to access it? Please PM me your KC VM IP and I'll take a gander.

rpqu · 1:43PM

It's so over.... php, node...

MannDude · 1:49PM

Ah yeah, so ticket submission probably broke due to today's earlier PHP patching.

Reviewing and will update shortly.

ChrisMiller · 2:12PM

KS is working fine. So is the IncogNet portal.

I'm connecting to the US using my KS VPN without issue from where I'm at in China.

MannDude · 2:14PM

Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)
Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.