New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Why is each server a standalone box with no failover connected to a $400,000 storage array?
na tbf I feel much more confident about a service I use when they are extremely transparent when it comes to downtime and the technical detail.
HAVE YOU EVER CONSIDERED CEPH?
HAVE YOU EVER CONSIDERED USING GOOGLE CLOUD FOR THE WHOLE OPERATION AND RESILLIENT BLOCK STORAGE FOR ALL DATA?
HAVE YOU EVER CONSIDERED GETTING HUGE CAPITAL INVESTMENTS, BUILDING SOMETHING COMBINING EMAILS, AN OFFICE SUITE AND AI (SOMEHOW) TO CREATE THE SMARTEST EMAIL SERVICE THAT THERE IS THAT NOT ONLY IS UP ALL THE TIME BUT ALSO REPLIES TO YOUR EMAILS FOR YOU?
Here's my opinion on the subject of reliability/redundancys, in a ideal world email should be hosted in a environment with distributed storage and redundant systems for everything incoming and outgoing.
But unfortunately this isn't the Idea world, unfortunately in the real world sacrifices need to happen due to mix of previous stack design decisions and pricing.
Going fully high availability is going to require a complete new mail hosting stack and infrastructure which is going to require significant investment which will result in significant cost increases for all customers, which I doubt most customers won't be willing to cough up.
The other reason why Mxroute isn't HA is simply due to the pricing segment that it's in which budget end of the mail hosting business low pricing with unlimited number of inboxes and in the case of reseller accounts which as far I know got no limit on number of sub accounts in a email industry which is common with per inbox pricing.
I think @jar current plan is the best option cost effective wise backup setup that allows to bring the system back up within a few hours and restore files later.
Hi @jar , I haven't used your product yet, but surely next year I will subscribe to MXroute after this post.
On a serious note: Biggest lesson here IMO isn’t that JetBackups hid an important option or using rsync, the biggest lesson is to just regularely test your backups.
Even though it’s tedious, boring and makes you wanna die.
(It’s been a while since I tested mine.)
Don’t tempt @jar with another migration.
Another Veeam recommendation from me, too. It's super flexible in letting you use your own hardware to store backup copies in multiple locations. This lets you buy storage from several low cost providers and achieve a high level of redundancy without spending a fortune.
Restoration time is still an issue with large datasets, so it's beneficial to have a local copy in the same datacenter and 10 Gbit ports between production and your backup repositories.
EDIT: This recommendation may not apply to Jarland's specific setup, and wasn't intended to. But anyone reading this thread and wondering about their own backup needs might benefit from others' recommendations and ideas for protecting their data.
A little while back my backup server's drive failed at ikoula. Honestly, compared to u @jar i didnt panic at all . But In the time it took to put in a replacement/swap order for my backup server, the drive in my production server at ikoula too failed. This was within days of each other. I almost had a heart attack then bcoz there was no backup of the backup.
With some crazy luck i managed to get 99% of data out of the drive via ddrescue. The point I guess is things break, no matter how much preparation one does. If that wasnt the case there wudnt be any fukushima, 3 mile island or chernobyl. Dont be too hard on urself. U at least have a backup system in place.
Yesterday I found out one of my 2TB SSDs died in one of my dedi idlers. There were 0 bytes written to it for 6 months (wasn't even formated or mounted). It just died on its own.
All things will eventually break, even idlers.
When I look for a provider I look for honesty, effort, and actions to prevent future incidents. Checks all 3 boxes.
This is expected behaviour for a server with a female name.
Jar you sir are one of the good ones, so keep doing what you're doing. The community loves you and supports you.
Anyone got a problem, send them to me.. lol
PS send all bobs receives on your networks to me Sir.
Where do you get your Veeam licenses, and how does your backup stack look? Like you install a server on one, client on the other server? I know nothing about it, so total newbie questions.
Their website being overrun with ways to contact them instead of an order button I can easily find, is usually instant no from me because “contact us for pricing” always means they’re ready to quote me a house. I see a free trial but unclear what exactly it’s a trial of.
If you're looking at Veeam, might as well have a look at Comet Backup. Pretty flexible, well priced and documented.
You can actually run community edition free up to 10 servers.
Here's a quick overview, but I'll explain the setup I just created.
In most setups I use, we'll have a backup server off domain and RAID6, or JBOD on a large volume. Thought being: If the backup server fails, you can kick-start a full backup after quickly restoring the config.
If you don't have ESXI or Hypervisor level connectivity to hit the VMs direct, you can use the Veeam Linux / Windows Agent to point to your machine (repository) usually referred to as the B&R server.
For best practice, Windows 2019, ReFS and Enterprise SSDs will get you where you need to be. They have a racket where you need to buy certified disks, but we got very cheap new Intel Enterprise 4TB SSDs that were certified for Win 2016.
For redundancy, you can create a 'Backup Copy' job which places zero extra strain on the original server, just leverages the existing backups on server to send them off to Wasabi, B2, or another low cost S3 compatible provider.
You can even create a tiered-storage type setup which I prefer, which keeps a week or so on SSD storage and 2 week / monthly on HDDs. Same deal with a backup copy job, just point to a different drive instead of external provider.
After one 'full backup' you'll have incremental that run nightly, and should things go bad, and your B&R server is virtualization capable, you can instantly recover and boot a machine to Hyper-V or vSphere, finalizing the storage during a maintenance window later, including machines backed up via the Linux agent.
DM if any other questions about the setup, have optimized a few companies that went from 8 hour recovery to ~40 minutes restore to ESXI. Really just took ReFS, SSD, and a few optimizations on the B&R server.
I think a few of the companies I know go through Ingram Micro for licensing, but Veeam actually offers the VMWare / Hyper-V essentials product for $7.13/host/month.
They do pretty good compression, and can go deeper with more CPU usage.
At older jobs, I've gone beyond backups and have migrated entire bare metal racks to VMs with this, it's never failed. At most, a line needed to be edited in grub to deal with the console differences so I could type. Once setup, it just works.
Anyone have any good setups of IMAP based email backups (pull based) and doesn't rely on SMTP based forwarding?
mxroute joined my list of providers to try for mail services, kudos for the explanation.
@risharde
Like @Kris mentioned, the community version allows you to run Veeam on up to 10 systems. It has virtually all features of the paid version, though comes with no support. I have a few important systems to back up, so I buy licenses in packs of 5 costing about $400 per year, purely to get support if I need it.
My backup stack consists of three parts:
One Veeam Backup & Recovery (VBR) server
This is the system that runs the UI console and orchestrates the backups. This part runs only on Windows, but you can use any Windows version (no need to use Windows Server). The VBR component can run on a VPS, preferably on the same network with the systems to be backed up.
Multiple Veeam for Linux agents
The Veeam agent software is installed on the systems to be backed up. I don't use Veeam to back up VMs at the hypervisor level in the way @Kris explained, but it excels at this if you're using Hyper-V or VMWare (which I don't). The agent software supports block cloning, so it monitors the system for changes on disk and backs up only the deltas.
Multiple repositories
In the Veeam world, these are your backup destinations. A repository can be a physical or virtual server in which you install some very lightweight Veeam components, or even S3-compatible object storage. I know you've sworn off XFS, but Linux using XFS is the cheapest and easiest setup for a repository. XFS supports fast cloning via the use of reflinks, and Veeam takes advantage of this feature to merge incremental backups into synthetic full backups very rapidly. Repositories can be located anywhere as long as they are reachable by the VBR server and the client to be backed up. Like I mentioned, I use cheap storage to hold my backups in geographically diverse locations. While enterprises may set up expensive multi-CPU systems with SAS disks and hardware RAID for their Veeam repositories, I find this overkill unless you need absolutely top performance. You can run a Veeam repository on a simple storage VPS with 4 GB of RAM, and then use backup copies to ship that data off to other repositories. I maintain TBs of backups this way.
Yeah, I can totally understand your reaction. Fortunately, this is a product in which the sales/marketing and technical/development departments are very different. Once you get past the marketing cruft and buzzwords, you'll find that Veeam is a very well designed and supported product.
@jar Would purchasing one "BF 2023 Small" support you? I renewed a small one from a reseller a month ago, but I would be happy to buy it if it helps you.
I think you did great. Thank you @jar
Keep up the good work @jar!
Life is such ... shit happens... but you are a man with a clear conscience and true intentions ... I suggest you go for it, even if you have to jack up the price towards a competitive price range, and I will still be your customer/reseller and support you all the while.
Thanks once again for your good work & best wishes.
Thanks for the transparency. I've never had a single issue with mxRoute, except for the "million or so" I lost by observing this thread, but it's great to see what I can expect if I do.
As an enterprise network guy, I've been through my fair share of challenging outages. Even on supposed "HA" hardware that is "never" supposed to outright fail. Screw the armchair opinions of what you shoulda done, those who put their full faith in one thing or another will eventually see their day too.
Kudos for the eventual full recovery.
I used to use Veeam primarily, and it was awesome for full-host backups - compression and incremental backups are great. But, recently got into Borg with Borgmatic to a secondary local storage target for backups of databases and filesystems separately and then monitor with Healthchecks.io. I did this because for what I run, I do not care about having full-host backups or even full-disk snapshots so Veeam didn't make sense for me for those cases. Borg handles massive backups by only backing up actual changes.
I appreciated outreach from @host_c offering help with backup space. While I don’t currently have a need to take them up on it, I thought public credit for offering to help is of value.
Zawinski's Law of Software Envelopment, also known as Zawinski's Law, states:
Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Oh wait, MXroute can already read mail, so it doesn't need to expand further.
By the way Lucy is a fossil's name in Ethiopia. Calculated age 3.2 million years.
Now, catch-all is disabled...?
Have a couple accounts with these guys. The downtime sucked, don't get me wrong. But the timely status updates alone made it bearable. Plus I personally know how infuriating it is when things go wrong and you're waiting on technology to keep up with how fast you want to work. Ended up migrating some clients to Exchange at my recommendation but I'll keep using these guys without a second thought.
Coming out of the woodwork for my once-a-year LET reply:
For other hosts, this should be the biggest takeaway of this thread. The force behind Jetbackup could be best described as "a cartel": there is no interest in a quality product, there is only interest in collecting money.
If you presently use Jetbackup, and think this situation won't happen to you: please prepare to write this same thread at some point in your use of Jetbackup.
Signed,
Someone who got blackballed from the cartel because he wouldn't commit to inferior quality
Good to see you back for your annual response.