Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


MXroute failed, and I'm sorry - Page 5
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

MXroute failed, and I'm sorry

1235»

Comments

  • emghemgh Member, Megathread Squad

    @Damian said: Coming out of the woodwork for my once-a-year LET reply

    Thank you for your annual LET reply.

  • sorry to necro as im late to the party but i am still recommending people use the service, even up until 10 minutes ago. went looking to see if there was any deals to help somebody out and found this. will definitely keep our services with mxroute in future. shit happens friend, the important part is how it's handled and i consider this the best it could be

    Thanked by 1rafaelscs
  • BikeHelmetBikeHelmet Member
    edited March 2024

    I guess I'm a necromancer too.

    @RDX said:
    I’m a long time and quiet user of mxroute. The money I paid isn’t worth the awesome stability I had. Servers fail and you are not a mega business to have all the fancy HA stuff (and even mega corporations have long down times once in a while.) It’s an unfortunate situation both to you and affected customers. But still thank you for doing everything.

    I think the clear communication is integral. I remember a few years ago when Hotmail had an outage (was that 2016?), and email vanished for tens of thousands of people. They got it solved... accounts were up and running again in a day or two, and people reported that their emails showed up again after about 2 weeks.

    So... if you're on par with hotmail, you're not doing terrible, right? Granted, this was the free service - but if your yardstick is Microsoft, you're not doing that bad.

    @Kris said:
    For best practice, Windows 2019, ReFS and Enterprise SSDs will get you where you need to be. They have a racket where you need to buy certified disks, but we got very cheap new Intel Enterprise 4TB SSDs that were certified for Win 2016.

    It's worth noting that back when Win10 allowed you to format to ReFS (2016 or so - I think it was removed a few years after that), I got some new storage drives for my home PC. I set one up with ReFS and used it to store web browser profiles and large downloads like linux distros and recovery tools and VMs.

    That drive slowly picked up filesystem errors despite never having anything show up bad in the SMART data. I was aware that things were going weird/wrong, and had all data in 3 locations. Eventually it corrupted itself completely. Chkdsk just refused to run on that filesystem, with a snarky message that for ReFS it was unnecessary. Microsoft did remove the ability to format to that filesystem on Win10 Pro, so maybe there was a bug or underlying reason for that, but the lack of available repair/recovery tools once things went wrong left a bad taste in my mouth. I no longer dabble with fancier filesystems for live data. I zeroed out the drive and reformatted to NTFS and it has been functioning fine under the same workload ever since...

    @Damian said:
    Coming out of the woodwork for my once-a-year LET reply:

    @jar said: fuck JetBackup

    For other hosts, this should be the biggest takeaway of this thread. The force behind Jetbackup could be best described as "a cartel": there is no interest in a quality product, there is only interest in collecting money.

    Last year I agreed to help a client with an ornery WordPress site that would no longer open, hosted at GoDaddy. I don't work on a lot of websites, as I'm more into other types of IT, but I agreed to help them because I know them, and I do have the skills despite not enjoying that type of work. It was a real estate site for a luxury villa. It's pretty common for many-million dollar properties to have their own websites. Anyway, it was not loading, was absolutely plodding, you could not access the wordpress dashboard, could not properly access the GoDaddy management pages or CPanel (timeout), and FTP would also just timeout if you tried to download any file.

    GoDaddy was... unhelpful. "Something is wrong" (yeah, no kidding), so their first level of support could not get me a cohesive backup of the files. They suggested deleting it and rebuilding the site in their page builder, and offered a credit for the inconvenience. But why would I stick with them if I had to rebuild from scratch? This is not the first time that I have personally witnessed such GoDaddy problems.

    I instead fiddled over FTP and discovered that I could rename things and move things, so I started chopping away at the site. Maybe a bad plugin took it down, or a forced upgrade or something? Maybe GoDaddy was allocating 1MB to I/O and some coalescence of glitchy software demanded far more? I cut away at themes, plugins, etc. - eventually I got a 90% broken wordpress login panel to come up, only plain background and text, no images, and ages after logging in, I could never get past the broken dashboard homepage.

    But, it indicated a performance improvement - and FTP started to allow me to copy files. So, I told it to copy the whole site over to my computer and left it running overnight... and the next day.

    18 hours later I had the site backed up to my computer - a real whopper, over 300MB large. (Wow!) I then uploaded it to RackNerd (took about a minute) and unpacked it. Then remembered that I needed the SQL DB. GoDaddy's Cpanel sort-of worked now, and by the next day I had a SQL backup, though it took a while to get the pages to open and downloads to start.

    My hunch now is that the server was going through some sort of death spiral, but at the time I just thought that they had strangled the I/O far too much.

    I tried for a backup again, but that never completed, so I instead manually restored the SQL backup and the FTP'd files and renamed things back into place.

    What greeted me quite surprised me... a Wordpress 4 website that had never had a plugin or theme update, and was pretty broken on modern PHP. I had to fiddle with many settings to get it loadable, but RackNerd had the performance needed that even with heavy I/O use (for some reason), I was able to use the WP Dashboard to start fixing the site, referencing error logs over FTP and patching things as I went. Most of the website was so dated, I needed to track down intermediate plugin versions and whatnot and incrementally upgrade PHP, plugins, themes several times in stages. It was a real mess. And all because the person that built it never saved a backup copy... oy.

    Eventually after much work - probably 15 hours spread across many days - I had a properly updated/patched website running on PHP 7.4 with the latest theme and plugins. With that, I went to sleep.

    I was very busy the next day, since I had lost time to this unexpected job that kept growing in scope, so I spent the whole day (a long 14+ hour one) doing callout IT work from the wee hours of the morning to well into the evening.

    The day after that, a weekend I think - although who keeps track of when those even start? - when I woke up, I found the site utterly broken. Critical errors galore. I had a SQL backup... from WP4, before all of the manual fixing/migration/etc.; I had not yet downloaded a new backup after finishing all of the work. I didn't want to repeat all that effort, so I figured:

    Well shoot - RackNerd has backups, right? And I know it had almost two full days untouched before it melted down. Dailies should catch all the changes.

    So I went to JetBackup, and tried to restore, and...

    ...it failed.

    They had to escalate and send troubleshooting info to JetBackup, but the gist of it is that the backups did not work. Some plugin had done something during the night and screwed up the SQL tables and its WP plugin files, and somehow the backups from just before that were unable to recover my files and SQL.

    Well that's a bummer.

    So I redid the whole thing while it was fresh in my memory. And then turned off and removed any non-core maintenance plugins. Where did that weekend go, anyway?

    @jar said: fuck JetBackup

    Hehe, I'm generally pretty polite, but I won't contradict that.

    Always test your backups! It's 100% the moral of all these stories...

    -BikeHelmet

    P.S. I don't fault RackNerd at all for that. It's a problem for the software developer to solve. In a messy situation, you just appreciate having a complete picture of the situation. RackNerd was incredibly prompt with their responses, identified the problem right away, kept me in the loop on their communications upstream, and most importantly were clear/concise. (Which helped me with the decision to redo the work. It didn't take as long the second time.) Those are great aspects for a business, and MXRoute has the exact same traits. When shit hits the fan, you need someone that talks straight rather than sugar-coating it, or giving you support as exceptional as GoDaddy.

    "Something is wrong. We can't get the files." ??? "The backup won't complete."

    I mean, that is clear/concise, but how about escalating it or explaining what the possible causes might be?

    /sigh

  • kevindskevinds Member, LIR

    @BikeHelmet said: Microsoft did remove the ability to format to that filesystem on Win10 Pro, so maybe there was a bug or underlying reason for that

    Windows 10 it was moved to Pro for Workstations and Enterprise editions.

Sign In or Register to comment.