New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
teh 1776 cult. supreme!
Pennsylvania now in stock. Pre-orders delivered ahead of schedule.
Just realized the original offer that this thread was created for was not available in PA.
It can be ordered in PA here: https://portal.incognet.io/store/limited-promotional-packages-and-plans/pa-4gb-ram-40gb-str-4tb-bw-4gbps
Thank you for the help, no account sharing/selling here, just my own stupidity as explained in the ticket
Grabbed myself the deal again in the "newly" added pensilvania location, heres to another 4 years ! (8GB for 8 years deals incoming lol)
Any news on this?
I forgot. =\
I'll see what I can whip up soon. VirtFusion is proving to be confusing to many. Have even had a cancellation or two because "idk what to do or how to use this"
No probs, I remember
@MannDude Any news?
Domain registration reopening??
Soon. Aiming for within the next week.
The jist of it is this:
WHMCS isn't a great platform for domain registration. I'm still auditing all of the imported TLDs, removing the ones that don't have any existing orders, and adjusting pricing across the board.
Kind of hard to explain unless you have had to deal with their system before.
For example, say you're a reseller like we are for most domain names. You can import a list of supported TLDs from the registrar, in our case, InternetBS.
You can have WHMCS auto-create pricing for these TLDs based on the pricing of reseller account with a markup in the form of X dollars or X percent. (Ex: +$5 or +20%)
Several issues occurred because of this. Many TLDs were imported that we can not support, such as random ccTLDs, either due to requiring PII that we can not supply or because they require registration fields not present in the module, causing API errors when trying to register.
Additionally, if the imported pricing had a "first year" special on the TLD, then all the subsequent years were based on that first year promotional pricing in WHMCS even if the 2nd, 3rd, 4th+ years with the registrar were much higher. A handful of lucky customers got multi-year domains with us taking a loss. There was really no way for us to audit it even after I quickly discovered it, as there were hundreds of TLDs and you'd have to use WHMCS' annoying system to manually check things one by one.
So what I have been doing now:
I exported a list of all domains we've registered that are active and took note of what TLDs people are actually buying.
I'm manually removing a ton of unused TLDs that have no interest by anyone, most of which were imported by mistake but there is no easy way to remove these once imported. (Have to scroll down / ctrl+f it on the page, check it, click the red remove button. Page reloads, rinse and repeat a hundred+ times. No way to do this for more than one at a time, and although the TLDs can be removed via a SQL query, this is dangerous because their pricing is still set in a different table in a very nonsensical structure. Safe way is to just do it manually.
Additionally, there is some weird DNS related issues and we may swap out our DNS management system. We're using the ModulesGarden PowerDNS setup ( https://www.modulesgarden.com/products/whmcs/dns-manager ) but there are issues with it, it's large and clunky, and paying $400 every year for it is overkill when it's as confusing as it is for end-users to use. It's not incredibly intuitive.
And finally, I've just been so overwhelmed with work and life that not manually reviewing the order queue and processing new domain orders everyday has been a nice relief. It's busy work that distracts from other things.
TLDR: It takes about 1 minute roughly to update a particular TLD, because I need to reference our pricing and remove multi-year options and reference what is actually being used by customers. Then I need to repeat this process about 500 times.
I'm about... 20% done but haven't been incredibly motivated to tackle it as I wait for more important things like new WA stock and the NL migration.
IF YOU ARE ON THE HOST NODE, "ALLEGIANCE"
Sorry for the inconvenience. We've never experienced anything like this before across dozens of nodes and I'm being told to expect the worse by those more capable than me to try to recover everything.
TLDR: Took a node offline to do a RAM upgrade so the node could actually be utilized fully. Expected this to be a quick window of maintenance. Turned into a no sleep, "shit has hit the fan" worst case scenario situation.
WHMCS is ass. I've heard similar configuration limitations/nightmares with WHMCS by other providers.
Data loss is unfortunate but is also a lesson for clients to always maintain their own backups (and that RAID is not backup). Any drive can fail at any time regardless of provider reputation.
Best of luck with both endeavors!
Thanks. While we have a variety of different monitoring things in place, looks like this node didn't have anything setup to alert us or warn us of drive health issues... So, as much as I'd like to just throw my hands in the air and say this was unexpected (which is true, it was), it's still possible that it could have been detected earlier. Tests ran back in March were all fine, however.
In any case, we're going to improve our hardware health monitoring and alert system as well.
I know we catch a lot of flack for slow support, but one thing I always did take pride in was having a pretty thorough monitoring/alert system setup so that if anything actually critical or noteworthy occurred there could be fast response. Will be reevaluating the setup to see how this can be improved as well.
Mind sharing what motherboard and NVMe drive models? PM me if you want to share. I'm just curious since I had the same issue in the past.
@MikeA PMed.
Recovery efforts are still underway, of course. I got the Crunchbits A-team on it.
I'm only announcing here because there isn't a way for me to inform only those impacted by email (yet). I love VirtFusion but it's missing some features I enjoyed from Virtualizor like being able to email users active on only a particular host node. I think this node was mostly LET promotional users as well, so this is the best I can think to do while half awake and the WHMCS service issue page.
Shit happens, even if you think you did absolutely everything right (and quadruple-checked everything), Murphy's Law will find a way. At least know that "big name providers" don't always have it good, like OVH's infamous datacenter fire and their "activate your disaster recovery plan" meme.
Glad to see that you're motivated to address deficiencies and work on improvements. Live and learn.
Not related to server hosting, but I was working on a task with a friend (who doesn't post on LET but does buy from providers here) and we were reviewing some data that we had to feed to a third party API. We had only one attempt to get this right with zero margin of error (irreversible damage if we fuck up), so we each individually checked the data at least three times over the course of a week. He eventually became irate after saying "yes it's all good" a dozen times.
Well, we fired off the data, and 5 minutes in we realized we botched the input. The culprit was Microsoft Excel being unable to properly render 64-bit integers, so opening a csv file with 64-bit integers and re-saving it with Excel will truncate some of the digits. We didn't realize this happened until we saw that we were getting the wrong results back from the API. Upon further investigation, we saw that all the 64-bit integers ended in 000, and eventually traced it back to this "quirk" in Excel. We didn't ever consider that a tool that is used all over the planet would corrupt our data like this.
I can't share specific details on what exactly we were doing, but the point is, even after exercising our maximum due diligence, we still screwed up hard. It really can happen to anyone regardless of level of expertise or competence. I thought our system was quite robust, but this experience showed that there's still more to improve on.
Working with excel the first thing to check the locale.
Oh, turned out I am on the affected node but luckily there is no sensitive data on the box. Just another reminder to keep updates or at least to utilize snapshots. Hope you will recover soon.
Reinstalling the node, but it's showing the incorrect amount of RAM now.
When it rains, it pours.
Datarecovery was a bust. Recovery tools (ex: HDD Raw Copy, DMDE, TestDisk, ddrescue) were unable to read or image either disk.
Only option now is to reinstall the node.
Well, taking into the account the gravity of the situation and time already spent on it, we may need to just swallow this, learn the lessons and move on.
@MannDude I really appreciate that you're not staying silent about this or giving us some half assed PR announcement but give us regular, transparent updates. I know it’s a difficult situation right now but I believe in your ability to push through and fix this.
Believe me, as much as I understand its frustrating for a customer to be without their VM... We're without a node at the moment. Waiting for the DC to check the RAM issue.
On the bright side, this is a LTO node and the hardware replacements are not out of my pocket.
Oh, props to @VirtFusion for implementing the feature to mail users based on their assigned hypervisor. This feature was missing before. Something I really liked about Virtualizor was the ability to send an email informing users of a planned maintenance window or migration or something based on their host node / location. Now I can do that with VirtFusion, too.
Shoutout to @MikeA who pointed this out to me via PM.
Had I known this feature was implemented in the recent update I'd have used it earlier.
Still waiting for the proper build to get racked so I can get things back up for those impacted.
When I wait, you wait. Its 2AM my local time so will get some rest and check back later.
Mr incognet getting eepy 😴 good night diva
Aight, an email went out 40 minutes or so ago with how to restore your service.
Status page shows the RFO. I've never really had to write anything like this, since it's the first time we've had anything noteworthy occur:
obligatory 'how do i enable disaster recovery plan i don’t see it in control panel' comment
I am back online
but I do not see any Backups section where I can make a snapshot, where is it?
PM me your server IP and I'll take a gander. Should just be here though:
I'm running a backup on my dev / test VM and it appears to be working. It's possible the feature may be disabled depending on what plan someone has, though. I'll be happy to check for you.