Incognet Kansas City Outage and WHMCS 403

MannDude

@TimboJones said:

@averagedatahoarder said:

@MannDude said:
Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

• First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)

• Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.

Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.

Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.

You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.

Corrective action was taken.

Literally restored the deleted v1234-xxxxxxxxxxxxxxx.raw from the filesystem with minimal corruption.

Anyhow, I think we both learned a lesson:

• "Measure twice, cut once" (for me). I'll reaudit the node later when I have time, it's apparent that the action for deletion was meant for the VM in the list either directly above or directly below OP's.
• Have offsite backups for important stuff (OP)

LEBUserJoe

@MannDude said:

@TimboJones said:

@averagedatahoarder said:

@MannDude said:
Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

• First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)

• Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.

Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.

Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.

You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.

Corrective action was taken.

Literally restored the deleted v1234-xxxxxxxxxxxxxxx.raw from the filesystem with minimal corruption.

Anyhow, I think we both learned a lesson:

• "Measure twice, cut once" (for me). I'll reaudit the node later when I have time, it's apparent that the action for deletion was meant for the VM in the list either directly above or directly below OP's.
• Have offsite backups for important stuff (OP)

• Don't have backup's auto delete and have a grace (incase)

drShank

When is it expected to be back online?

Fourplex

Right now, we are working on it. We are rebuilding the database and WHMCS on another host.

We fortunately got the customer and service information.

gbzret4d

@averagedatahoarder said:

@yoursunny said:

@MannDude said:
I'd like to offer $500 as compensation.

Plot twist: it’s account credit.

Plot twist: expires after 30 days

Not eligible for discounted or special offers

Fourplex

@gbzret4d said:

@averagedatahoarder said:

@yoursunny said:

@MannDude said:
I'd like to offer $500 as compensation.

Plot twist: it’s account credit.

Plot twist: expires after 30 days

Not eligible for discounted or special offers

We'll honor existing coupon codes in the migration.

JohnFilch123

any deals to celebrate the recovery?

averagedatahoarder

@JohnFilch123 said:
any deals to celebrate the recovery?

$7/year please and thank you

TimboJones

@MannDude said:

@TimboJones said:

@averagedatahoarder said:

@MannDude said:
Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

• First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)

• Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.

Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.

Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.

You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.

Corrective action was taken.

Literally restored the deleted v1234-xxxxxxxxxxxxxxx.raw from the filesystem with minimal corruption.

That's not corrective action, that's containment.

Anyhow, I think we both learned a lesson:

• "Measure twice, cut once" (for me). I'll reaudit the node later when I have time, it's apparent that the action for deletion was meant for the VM in the list either directly above or directly below OP's.
• Have offsite backups for important stuff (OP)

Corrective/preventative action is doing something like having your work double checked or the obvious, delayed permanent deletion. Something changed so this doesn't happen again. "Measure twice, cut once" doesn't qualify, it's the same human failure that led to this issue.

MannDude

@TimboJones said:

@MannDude said:

@TimboJones said:

@averagedatahoarder said:

@MannDude said:
Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

• First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)

• Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.

Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.

Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.

You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.

Corrective action was taken.

Literally restored the deleted v1234-xxxxxxxxxxxxxxx.raw from the filesystem with minimal corruption.

That's not corrective action, that's containment.

Anyhow, I think we both learned a lesson:

• "Measure twice, cut once" (for me). I'll reaudit the node later when I have time, it's apparent that the action for deletion was meant for the VM in the list either directly above or directly below OP's.
• Have offsite backups for important stuff (OP)

Corrective/preventative action is doing something like having your work double checked or the obvious, delayed permanent deletion. Something changed so this doesn't happen again. "Measure twice, cut once" doesn't qualify, it's the same human failure that led to this issue.

I don't disagree, but in the case of things like delayed permanent deletion, off-site snapahots (if user had created any) not getting axed at the same time as the source VM, etc aren't really features available in the legacy panel (Virtualizor).

I do like the idea of snapshots being stored for a period of time after termination, it's something I want baked into what we're building now... But need to find a balance between "OMG INCOGNET KEEPS UR FILES AFTER YOU CANCEL" and "Ah, I cancelled the wrong VPS by mistake can you restore?" ... Probably just have it as a feature the user can toggle themselves so they can choose if we retain backup data for a week or a month after deletion, whether it's their doing (cancellation, non-payment, etc) or tired staff (me) making a mistake.

I guess there can also be a feature for any manual deletions to require two staff to agree. One can mark for deletion and the other can confirm the selection is correct and accept it, but my fear would be simple complacency where "everytime I check, you're right" and just blindly agreeing and then one time approving an accidental edge case.

Gives me some stuff to think about in any case. Obviously we gain nothing from mistakes like this so want to implement some mechanism or strategy to prevent again.

JohnFilch123

@averagedatahoarder said: $7/year

1776

TimboJones

@MannDude said:

@TimboJones said:

@MannDude said:

@TimboJones said:

@averagedatahoarder said:

@MannDude said:
Ok, this did not take too long to review before I found the problem... and it's a big one.

Now, I could have gone about this in two ways.

• First way, I lie. I blame something else. It’d have been easier and likely would have helped me save face. (Disk failure, corruption, breach, new guy, etc)

• Second way, I tell the truth and own up to an honest mistake that I've made.

I’m choosing the second way even though it’s going to cost me more than what I’m going to offer as compensation for this incident.

I fucked up. It appears I've removed the VM by mistake.

The last two days I’ve been auditing our legacy systems and having to manually terminate accounts that should have been done automatically. We had a little more than 300 legacy VMs left that have not yet been migrated to VirtFusion yet, and between the constant Virtualizor issues (and sometimes WHMCS issues) I’ve closed the communication gap between the two so the usually automated tasks have become manual tasks. With the security history it seemed… safer this way, at least to me, at the time...

However, I do see in the admin log that I terminated the VPS, and I see in WHMCS that it did NOT qualify for manual action. (Most terminated servers from the last two days were things that should have been removed a month or more ago but was still live and online)

I'd like to offer $500 as compensation. I did check the offsite backups for any snapshots, unsure if there were ever any made (they’re user generated, not automated) but as by design snapshots are removed when the VPS is terminated. I checked anyway, because Virtualizor systems are often broke, perhaps they’d still be there… but they are not, if they were ever created in the first place.

He was paid up for another week. This wasn’t a Virtualizor issue (though lack of confidence in the product is what caused me to take on the burden of manual audits / actions). This is 100% on me, an honest mistake that I made. The customer had no abuse issues. No high resource consumption. Wasn’t overdue on billing. I fucked up.

You'll need to implement your disaster recovery plan and restore (understandably, likely elsewhere) from your self-created offsite backups.

I am genuinely and sincerely sorry for this mishap, you, like me, and everyone else right now, are all in constant patch/repair/update mode. There is no good timing for bad news like this but I'd consider this to be a, uh, worse time.

Actually, at least personally/for me, this makes me trust IncogNET more. IMO it is much better to have an honest host than a host that will just blame it on the user, upstream issue, etc. I respect your honesty.

Why the fuck are you buying services from dishonest hosts? Honesty is a fuckin given.

You mean blunt or straight forward or something along those lines. A more professional company would have just kept it to three sentences of something happened, here's compensation, here's why it won't happen again. The pleading that "i fucked up" doesn't have corrective action attached. That's what you want in a host.

Corrective action was taken.

Literally restored the deleted v1234-xxxxxxxxxxxxxxx.raw from the filesystem with minimal corruption.

That's not corrective action, that's containment.

Anyhow, I think we both learned a lesson:

• "Measure twice, cut once" (for me). I'll reaudit the node later when I have time, it's apparent that the action for deletion was meant for the VM in the list either directly above or directly below OP's.
• Have offsite backups for important stuff (OP)

Corrective/preventative action is doing something like having your work double checked or the obvious, delayed permanent deletion. Something changed so this doesn't happen again. "Measure twice, cut once" doesn't qualify, it's the same human failure that led to this issue.

I don't disagree, but in the case of things like delayed permanent deletion, off-site snapahots (if user had created any) not getting axed at the same time as the source VM, etc aren't really features available in the legacy panel (Virtualizor).

I do like the idea of snapshots being stored for a period of time after termination, it's something I want baked into what we're building now... But need to find a balance between "OMG INCOGNET KEEPS UR FILES AFTER YOU CANCEL" and "Ah, I cancelled the wrong VPS by mistake can you restore?" ... Probably just have it as a feature the user can toggle themselves so they can choose if we retain backup data for a week or a month after deletion, whether it's their doing (cancellation, non-payment, etc) or tired staff (me) making a mistake.

Don't make it a setting, just put the days retention in your ToS. No surprises me but everyone gets the backstop.

I guess there can also be a feature for any manual deletions to require two staff to agree. One can mark for deletion and the other can confirm the selection is correct and accept it, but my fear would be simple complacency where "everytime I check, you're right" and just blindly agreeing and then one time approving an accidental edge case.

Gives me some stuff to think about in any case. Obviously we gain nothing from mistakes like this so want to implement some mechanism or strategy to prevent again.

Quality IS hard, but makes for a better product and processes. Easier sleeping, vacations, etc.