New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
what do you use for backups / DR for your hosting servers!
Jasonhyperhost
Member, Patron Provider
in General
hey guys!
interested to see what everyone using for backups & DR for there hosting and VM Servers, we have been using WASABI for about 2 years now very solid! , but im sure there other products that would do the job better such as colocating another server in a seperate dc ect whats everyone do!
Thanked by 1plumberg
Comments
Mostly Hetzner storage boxes. One per physical server.
Copy + Paste
We use a three tier setup.
Each server has a dedicated hard drive for a local copy stored for 7 days.
That hard drive is rsync'd over to two backup servers, and stored for 14 days.
That backup servers are then sent over to cold storage for 30 days.
Remember RAID is NOT a backup solutions!
Curious how you approach this: Do you backup your Hetzner servers to Hetzner storage boxes? And if so, do you use a different region for the box?
B2
@jar that dont soud too great, your surely not backing up hetz server to hetz storage boxes?
Backblaze @tjn ?
Rsync to my hetzner storage box (hot storage) and my local hdd every week (cold storage)
I'm continuing to backup to Hetzner as I slowly move to OVH, but right now I do prefer my backup in the same DC. A full server failure would take over 3 weeks to restore from backup if it were in another region.
Eh. I feel like this is one of those things that people with smaller data sources don't understand well. To be fair, neither did I until I had enough data to get here. Try restoring 4TB in 1800 DirectAdmin accounts with JetBackup across the world without everyone on the server doing a chargeback after the first week of repeating "The restore is running I swear." A backup that can't be restored in a reasonable timeframe isn't much of a backup. Building my own backup system to replace JB would be better but that's a big job, not statistically justifiable right now, and not something to just do just to make things sound better in a forum post.
There's always room to redo and rethink backup strategies, and mine is always under review. But if people are focusing in on my service that was designed to creatively cut corners to attack the common price points of the industry I'm in, I don't think they're thinking clearly if expecting me to have multiple geographically redundant copies of the data. Rather, they should expect me to pick facilities that aren't likely to see major events that take out the entire facilities (read: not shipping containers or other experimental building types) short of a nuclear event.
People would be utterly astounded at how many major corporations would fall if their entire data centers went up in flames with nothing being recovered. Best practice on paper doesn't always align on every issue. I don't think it's reasonable to expect me to have overcome every potential theoretical issue, but it's surely something I continue to work toward within the confines of the business strategy (meaning I won't triple prices to increase redundancy for a benefit of 1% in a particular statistic). At least count on me not lying about it though, as would be very common in this industry ♥️
Side note: I did actually try JetBackup recovery on a system using about 4TB, with 1800 DA accounts, I'm sadly not exaggerating. Their support says it's normal and not a problem to fix. Had they said otherwise, I might have replied differently today. Tomorrow may be different than today, who knows. Change is a pretty reliable constant.
Kopia, great piece of backup software
Yes! Great for 3rd location backups to be honest.
Immutable options, encryption at rest, and it's cheap.
If I need to download/restore anything I use Cloudflare in front, and it's $0 egress for as much as you can download.
Vs Wasabi, I don't have to worry about egress fees or the 90 day deletion policy - which is ideal because a lot of incremental backups I don't keep for longer than 35 days.
3 weeks!!?? That's insane!
I don't use JetBackup - actually looking at using it on a new DA install - but there's got to be a better way?
I use jetbackup @jar
Are you saying that standard DA or cPanel backup system might be better?
What type of backup you were using in JB, was it incremental or full compresss/uncompressed ? Sounds like incremental with many files, right?
Read @jar's post carefully. Four posts above this one. It has genuine insights and experience about real-world disaster recovery and business continuity planning.
I use borgbackup from all of my servers (a VM host just backs up config, VMs are backed up separately if important), to a number of destinations. Currently, everything gets backed up to:
A few of the most important are also backed up to a couple of old VPS that had a reasonable disk allowance.
I use different encryption passwords for each server and destination combo, so I also have to make sure I have a backup copy of those somewhere for when I want to restore things. I've tested out migration/recovery on a small scale when I retired my old 500GB KS-1 and recreated it as a VM on my KS-LE-1.
My backups are done to my local physical drives. It might be slow and old fashioned but it's safe and it works for me.
Incremental, not sure off hand if it's compressing but it is encrypting. The default DA backup system is still worse, one by one full account backups transferring every day is just too big of a job at this scale. JetBackup is the most reliable, well performing solution I've had to date but actually restoring it is an incredibly long event. Possibly made worse by maildir (but not necessarily justifying mdbox, which directly harms requests like the one recently made in my offer thread).
DA's backup tool is so poor, would be great if they could work on optimizing it ,but with DA only 3rd party ones are feesable , we use jetbackup too (Currently) and use wasabi storage its much better than the hetz and ovh storage we used to use, we will soon be moving to a Synology in a seperate DC works on our VM servers well , just need to grab new ones for our hosting servers
what is backup?
For personal stuff I use Kopia as software with iDrive e2 for the storage. At work Velero software (we use GKE) and Google Cloud Storage
i use iperius backup
Do the users/mailboxes become usable as they've finished transferring to the replacement box, or the WHOLE thing needs to be restored, restarted and then everyone is back up? In other words, some lucky bastard is the first account restored and has least amount of downtime vs everyone else.
https://www.rsync.net/ is very comfy and nicely priced for what they offer.
Yeah pretty much that, things come back one after another, so whoever is first gets lucky.
Though after my success with full migrations via rsync, the idea of keeping a hot spare server in another DC is more and more appealing. Too high on overhead today, but the market is always shifting. Of course by that time I could be moving toward the death of centralized storage altogether.
Hello Everyone,
With the upcoming release of JetBackup v5.3, we want to share a little bit of insight into some updates that may be relevant to this conversation. One of our primary focuses in JetBackup 5.3.0 is the optimization of the general DR process, part of which is improving our indexing engine. Utilizing a more efficient database management system and updated indexing process, we have observed a significant reduction in the system memory usage as well as faster reindexing of destinations for mission-critical restores and recovery. This means that restoring a DirectAdmin server with thousands of accounts and several terabytes of data will be substantially quicker in JetBackup v5.3 as compared to v5.2.
We want to thank you all for your feedback, and if you have any questions please do not hesitate to reach out!
Best Regards,
JetApps Team
We have setup an Acronis Cyber Infrastructure 5-node cluster in each of our locations, and use that as a primary backup solution (each node has dual 10G NICs, SSDs for cache+metadata, SATA drives for storage).
On Acronis we keep backups every 8 hours for a period of 3 days, 7 daily, 8 weekly and 12 monthly restore points.
The ACI cluster is on-site with our cPanel servers, allowing fast backups / restores.
For off-site backups, we use JetBackup 5 which stores 7 daily backups on Hetzner Storageboxes (different storagebox for every physical server).