New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
All services for us still (and will continue) to be MXRoute. Have other friends/family using you for their stuff. No problems. Not going anywhere. Hell, you could(should) probably charge us more.
That being said, it's one blind spot I've completely neglected and just tried restoring an e-mail backup that I thought was fine only to find out it, in fact, was not. Thanks to your public attention to this it forced me to check a recovery plan for our e-mails I thought was working only to find out it wasn't, thanks @jar.
I missed this, but I don't want to ever put our storage VPS servers out as infallible. We take them seriously, but it's still a large RAID-6 array and still possible to have a combination of events causing an issue.
@jar
Dude you are a intelligent and awesome person, when running a small business bad things happen, and while it does suck when it strikes, all of LET know who you are, you are a awesome and caring person. When this happened you could of said "well that's all folks" and just left. But being the person you and the person LET knows you not only continue your small business and acknowledge a fault has happened, you credited the users $25 and currently taken steps on what went wrong and trying to fix it and knowing you doing your best to make sure it never happens again. Dude you started this business, and continue to give awesome prices for great service that is mxroute. You are always truthful and honest. You don't have millions that other company's have and I am extremely happy with my $5 a year and $45 triennially service I have with you. You have not only a amazing but truly unique and very loving service when it comes to cheap mail server plans.
Listen in all honesty, you helped me when I was wrong in the past, and I truly look up to you, I am a reddit and lowendtalk lurker :-P , even if something bad happens 1,2,3,4,5 or more times god forbid in the future, I know not only that you do your best to fix what went wrong but will continue to do a amazing job. You truly care for your customers. You know how rare that is nowadays.
Keep being awesome dude and never change, many of us not only apricate what you have done, but see you as someone to look up to.
I only wish the best for you and your awesome company, but hope you expand on other fields and always grow.
I've worked for a couple of big tech companies, one of them stored passwords in plain text and the other got 1.2m wordpress installs hacked.
They blamed the customers when it was their error and you have blamed yourself when it's not your fault.
You undersell and over deliver. This honesty and humbleness only makes people more sure that MXroute is a brilliant service.
Good luck, hope everything works out!
@jar what happend has already happend and you will fix it so my question is:
are you accepting (non-reseller) customers buying credits at the moment
is this some kind of marketing strategy?
you fail to impress me
This sounds very stressful, but you'll survive in the end, as will MXRoute. Bad things happen to every host
Have you tried Borgbackup, Restic or Duplicacy (not Duplicity)?
You should at least take a snapshot (e.g. using LVM snapshots) before copying the data, so the backup is consistent. If you backup backs up the MySQL data while data is being updated or moved around (e.g. if
OPTIMIZE TABLEis being ran), the resulting InnoDB data can end up corrupted.On the other hand,
mysqldumpis not a good idea for large databases, since it locks the tables when it's running. A hot backup tool would be best, but I don't know if any free ones exist any more. There's an official MySQL hot backup product but it's part of their enterprise product which is $5000+ per license (thanks, Oracle).Running a secondary server like @HostEONS suggested is a good idea too..
If it is, it sure wasn’t worth it 🤣
Come on Jar..
Shit happens! Learn from it and move on!
Just added MxR to my signature.
You've mist the critical line "I've lost millions". What a waste of an opportunity...
@jar I'm in the tech industry for decades but not as a tech person per se, it is merely my hobby to enjoy playing with web sites and hosting. I have lurked heavily on the low end forums the past few years, and spent about $50 total for a half-dozen annual services. I became an MXroute customer on a BF deal last month, simply to tie together all my hobby domains' emails in one spot, because I was impressed with what and how you write.
This forum is a ton of fun, I get a year of joy for well less than the cost of an In-n-Out burger meal with my wife.
Perfectionism, and "let me prove my worth before I ask you to pay me a fair price" are wonderful for customers and for managers to exploit (haven't we all worked for those people?) They are a painfully un-necessary and expensive way to build a business.
As a relatively old guy in this crowd, I urge you and those other earnest providers here like you not to sacrifice your prime working/earning years on the altar of being the most perfect service provider for 10% of what lesser providers easily charge established profit-making customers.
What you have learned the hard way about email deliverability is nearly priceless, and you share much of it freely in your blog and here. Sure, you are also learning priceless lessons about how hard it is to backup email systems cost-effectively.
Don't let a few 19 year olds in their basements drive you crazy. Let's just say my particular tech-servicing field involves helping people who make millions a year avoid responsibility for their daily mistakes and failings. Bad stuff happens even when we are conscientious.
It's my honor to get your annual email service for $5, but honestly there are potential customers out there who could use you to make millions more in sales of their own products (without spamming), as well as professional firms that want to know their adversaries likely received the emails they pretend not to have received.
Those potential clients deserve your attention and skills, not us here -- we're just being entertained cheaply and like being given freebies.
Best wishes,
Deep Lurker
Is there any way we can help with backups, storage, etc? We are willing to help, please contact us to discuss details.
That's why we love MXRoute, if my other email provider would care that much.
Mxroute failed to deadpool, which is great.
Maybe start a separate transaction email service?
A few days ago. I said you can use dual redundant san storage.
I don't care about servers. We feel sorry for you.
We don't want to.
more tears. more gray hair. Sleepless nights.
I'm sure it will get better. And you'll make everyone happy (It's a lie and it's not possible)
Okay, now I can get 1 year free reseller, I think I deserve it
Let's lose millions of dollars together.
@jar ❤️
Any of us that have been around long enough have all been in your shoes at some point in the past and no doubt will be again in the future. Bad shit can always happen with hardware, don't let it get you down too much.
In terms of the backup/restore I've yet to come across a quick way to restore millions and millions of individual files across a network. I tend to spread things over smaller servers so if the shit hits the fan the servers can be restored in a few hours but the downside is your costs add up.
So yeah, chin up and try catch up on the missing sleep!
You didn't fail you just learned an expensive lesson, I personally noticed how much time incremental backups needs to be restored after few backups, thats why I stopped using incremental backups and jetbackups and stuck with the old fashioned zipped backups, I also noticed that hetzner storage boxs speed is slower from outside so as you said best option is to create identical failover servers to just switch to them if shit happens
Best advice I've seen on the thread.
RAID is wonderful until you get bitten, then something like a bad stripe happens, or disks cascade fail while trying to rebuild, but fixing that MySQL issue should have been easier.
Still, I do love MXroute, even if Jar and I don't agree on his spamassassin setup
I'm not even on lucy node yet I already lost millions because of this
This is most likely something that you are planning anyway, but just in case...
Whatever you land on for backups/DR going forward, make sure you give it a full test so you have a firm idea of likely restore times and to make sure something simple but highly inconvenient (like that off-by-default setting) hasn't been missed or set wrong.
Furthermore, repeat this test at least occasionally. For my home mail, document, and web, servers I have VMs (lower spec, not publicly visible) on another host that automatically hose themselves and restore from the latest complete backup daily. I check occasionally to make sure it is working and has recent mail (I should probably automate that a bit more). In a servers-set-on-fire situation I could bump up those VMs specs, do a more recent restore if they are too many hours out of date, and have them take over. My exact setup is probably impractical at your scale, but I expect you'll be able to arrange something similar at practical cost+hassle.
No problem, @jar. Appreciate the transparency and your willingness to prevent this in the future.
@jar, i fully trust our data are safe with you, bad things could happen and downtime is nothing compared total loss of data, so since you recover the data you deserve the trust.
thank you for providing affordable email service.
Great honest no BS writeup.
Since you're going the rsync route, this is a wrapper for parallel rsync operations, very versatile: http://moo.nac.uci.edu/~hjm/parsync/
Only other way I'd try backing this up is Veeam, with something like Wasabi/B2, or just LAN storage. Block level is needed for this level of tons of small files, but you've got it figured out after this hell it seems. It's just saved my arse more times than I can count, so obligatory Veeam plug.
Would really be refreshing if all hosts were this honest. I'll be getting a mxroute account for my UptimeKuma instance alerts, right now just on a free smtp2go account and nearly outgrowing it.
Thanks for sharing. Good read.
Failure do happen and you have demonstrated to have tried every single thing to recover the server.
Also a reminder for us all that we shall have our own backup, no matter which service or provider. This stuff have happened in many other tech companies. HW RAID tend to fuck.
@jar I also run a private mailserver with ~4m emails or 2 TB and the nightly backup with Borg (to borgbase.com of course) takes about 10 min. That's RAID 1 HDD and older E5 CPU. A larger SSD RAID shouldn't take much longer.
Also, avoid HW-RAID, unless you have ancient CPUs.
I also use Veeam, I use it for full backup of all my VM in production use, considerable faster than any file based backup I've used in the past.
I've had to do a full restore once you don't get the slowness/overheads of copying each file one at a time compared to a file based backup e.g. rsync, the limiting factor is how fast you can pull the backup and disk speeds.
I’m a long time and quiet user of mxroute. The money I paid isn’t worth the awesome stability I had. Servers fail and you are not a mega business to have all the fancy HA stuff (and even mega corporations have long down times once in a while.) It’s an unfortunate situation both to you and affected customers. But still thank you for doing everything.
@jar appreciate the full rundown here! It totally sucks when you're racing against time to rebuild a server / software! Wishing you all the best!
Wow - as a MXroute customer thanks for sharing. Thoughts going through my mind is MXroute needs a dev blog for these types of issues - something to read and learn from
Not sure if you're already using it, but rsync 3.2.3+ xxHash checksum support with zstd can be way faster https://blog.centminmod.com/2021/01/30/2214/fast-tar-and-rsync-transfer-speed-for-linux-backups-using-zstd-compression/. I'm always obsessed with making backups and restorations as fast as possible ^_^
Maybe look at tar + zstd too - tweaked, you can do zstd compressed backups at near disk line rate speeds as fast as tar uncompressed backups if you give up some compression ratio - balancing archive size vs speed
For MySQL backups, maybe look at Percona Xtrabackup or MariaDB's MariaBackup tool or at least a InnoDB binary log backup routine allowing you point in time backup and restores?