New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
lucy.mxrouting.net
Been offline over 24 hours so far.
Status page shows maintenance failed and likely a rebuild / restore necessary. This is dragging on and my systems are queuing outgoing mail etc so not panicing but if I used incoming mail I would be getting a bit upset (losing millions).
Comments
Indeed. Having to move nearly 8TB off of a Hetzner storage box to a new server is not the fast backup plan I had imagined. Even less so when apparently JetBackup keeps a very important option disabled by default, which hugely increases disaster recovery time if not enabled after setting up your backup server. So yay to that.
If at any point I sound relaxed about this it's because I've already cried, already screamed, and I've slept very little. I'm extremely sorry for the trouble, and I do want to change my disaster recovery plan after this, but that doesn't fix this right now and fixing this right now is the priority.
Sir, I am losing billions per minute.
Thank you for the feedback
@jar probably not sleeping atm. Believe me, that man cares a lot.
Just have patience and after services up ask for full post-mortem. And I mean FULL - like all cause of trouble (every tiny bit of info), what was done and what preventive actions taken to mitigate failure in future.
Also, you may demand generous compensation for your loses. Millions, billions or any other cosmos. All those goes straight to /dev/null.
So the compensation in billions for such server located within a galaxy goes into the hyperspace between the galaxies. This is interesting, as these billions could potentially be used to form another galaxy.
I believe the old saying "be careful what you wish for" would therefore come to pass.
We have a division that sells spaceships.
Each spaceship sells for trillions, which earns us profits on the order of billions.
During this email outage, there could be a sales inquiry that would lead to the sale of a spaceship.
Due to we not receiving the email in time, the sale could have gone to a competitor.
Hence, we are losing billions.
If you are - like you say - losing billions… you’re in the wrong price range for your email hosting needs, by several magnitudes. @Jar is great, and offers great services, but yeah he is doing his best.
We have $5/year budget for email hosting.
This time it wasn’t enough.
I wonder how a server called karen.mxrouting.net would respond to requests.
Don't be a dick! @jar is doing his best. You should be more grateful to have @jar in your life. How do you think it makes it feel when @jar hears such comments from you? Of course, if he has any brains, he probably doesn't care a wit about you. Just ignore these haters, @jar!
It takes a swig from its vodka-fortified Starbucks mug and demands to speak to @jar's manager.
Don’t they usually skip the mangers and go straight to the directors?!?!
Shit happens.
It will pass.
Stay cool
♥ @jar
I do appreciate the withholding of the beatings by most people up to this point. I'm prepared for them after.
We just too busy crafting paperwork for court. We gonna sue for billions.
PM me for all your ZFS needs. Any backup that relies on rsync,rclone,rthis,tthat is basically shit. Essentially it's not, until you need to restore
Wouldn’t even be that bad if it weren’t for the number of files. If they were packaged backups they’d transfer faster, it’s that every email is a file and the backups were incremental. Probably the only time mdbox shines.
Only people who never do any work can afford to never make mistakes (never fuck things up). It's normal.
The difference between good, and bad providers is that with the good ones problems are a rare exception, not the norm.
Let me know if you need any help - 'cause I'm on Lucy (just needed an excuse to say that, but I do mean it
).
Relja
@jar you got this bro!
Also wanted to say, maybe your operation is already massive and maybe moving some stuff to cloud with redundancy might be a good fit. A premium tier plan for those that cant lose millions.
The beating will continue until morale improves. Why are you leaving hetz?
I feel like I’ve answered the question differently every time I was asked, just because a different thought floated to the top first each time. I think really the biggest problem is that I like NVMe disks but they use really bad ones that die very quickly under the workload we do.
I'm on blizzard.mxrouting.net and not losing billions. Is this normal?
that is not normally, check him pc
loved this.
@jar is the GOAT
Every IaaS will fail, and not just once
Rest assured @jar - this will occur again. Maybe not the exact same issue, but something at some point will cause downtime and some other thing will cause the restoration to be deleyed
The difference between shitty providers and not just MXRoute but every reputable provider out there (AWS, GCP, Azure, even OVH and the likes) is that WHEN (not if) things fall apart, the good providers show that they care by explain why and making it clear what changes they’re making to make sure the same thing dosen’t happen again (for the same reason, at least..)
I thought all the servers were named after days of the week.
I think @jar ran out of days
A glance of the server names:
https://crt.sh/?q=mxrouting.net