New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Incognet Kansas City Outage and WHMCS 403
While Incognet is technically our rival, we do host a critical service there for redundancy (our WHMCS).
We have a Kansas City Incognet server and right now it's offline. Meaning we can't ping it or see it in Virtualizor. We have the Legay EPYC VPS.
And yet I cannot send a ticket to Incognet. I just get a 403 Forbidden when I try to send in one.
Could @MannDude or someone else check into this?
Comments
Portal isn't down, are you still unable to access it? Please PM me your KC VM IP and I'll take a gander.
It's so over.... php, node...
Ah yeah, so ticket submission probably broke due to today's earlier PHP patching.
Reviewing and will update shortly.
KS is working fine. So is the IncogNet portal.
I'm connecting to the US using my KS VPN without issue from where I'm at in China.
Ok, this did not take too long to review before I found the problem... and it's a big one.
Now, I could have gone about this in two ways.
First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)
Second way, I tell the truth and own up to an honest mistake that I've made.
I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.
I fucked up. It appears I've removed the VM by mistake.
The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...
However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)
I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.
He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.
You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.
I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.
ops!
Fugged
shit happens but it is great MannDude is open about it.
big whoops
hey i'll take that deal, delete away!
Lmao
holy 💀🥀 hopefully @Fourplex has backup
...
Lmao
I hope you backed up your database.
Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.
Important news, don't miss @raindog308
jesus fvcking christ, hope OP has backup
Plot twist: it’s account credit.
Plot twist: expires after 30 days
Well, that's the first (and I pray hopefully the last) time I've ever had to recover and recreate a deleted disk image file on a live system.
Too tired to write a long speech about it, been a hell of a last couple of days. Server is pinging again. Sites aren't up but could be a simple case of needing to kick Apache / Nginx. Even with some corrupted files, enough data should have been recovered to make this less bad.
Only comment would be, I think you should redesign for auto-deletion of backups for a period of X days to prevent this ever happening again, whether it's you or a future employee.
Otherwise your backups themselves cannot be trusted, otherwise shit happens
whoa...yeah full day of CVE remediation and the hits keep coming. good job though. definitely cool to see an honest provider trying their best
As an update, we were able to get our billing server back for the most part. We are working towards MariaDB recovery as there was some file system corruption.
Not everything is online yet, but thanks for @MannDude for helping recover our server!
so little drama.... do better next time.
@Fourplex @MannDude
Seems like an edge case. Really good of both of you to bring this publicly and handle it in a professional manner.
y-you're just going to restore from the offsite backups that you were keeping, right?
Could be handled in PM.
But its nice to see a good outcome.
He PMed me a few hours before this thread. But I've got 300+ unread inbox notifications. He tried, I just never usually look at my PMs.
Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.
You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.