All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
The Backup & Restore thread
I thought it would be useful to have a dedicated thread where to discuss backup strategies, tools and storage providers etc. Anything that has to do with backups and restoring data. Sharing our experiences might help others save some time when implementing their backup strategy.
I kinda review my strategy every 1 or 2 years, and I have been doing this these past few days.
For the onsite backups for my Mac I've been using Carbon Copy Cloner to back up to a USB attached drive, and TimeMachine to back up to a RAID-1 Asus NAS.
For offsite backups (both Mac and a few servers), until recently I was using Borg as tool (using a nice script I wrote) and NetDynamics in The Netherlands as storage provider. As secondary off site backup, I was using Arq with iDrive e2 first, then with other providers I have been trying with it, namely Storj, Synology C2 and Wasabi. I also have the most important data in Nextcloud (Hetzner Storage Share).
When I learned, to my surprise, that the storage I was using with NetDynamics was RAID-0, I stopped using Borg and started using Arq with both Storj and Wasabi (I got $25/mo in credits for 5 years with Storj thanks to some nice person who works there in appreciation of my open source work; as for iDrive, I cancelled after a day-long outage).
However I decided to try other tools again. Today I signed up with NetDynamics again for one month using the Finnish location since it's RAID-6 and I am in Finland, to see the difference compared to The Netherlands location I was using before. Backup was slow and restore very slow. I know that some people think that it doesn't matter if backups and restores are slow but for me it does. If backups are fast, I can back up more often. Also, if a major disaster happens and I need to restore from offsite backup, I don't want to wait for days to restore my data.
I like Arq's UI and it's battle tested and stable with years on the back, but it's too slow for me. Even with a Gigabit connection restoring just 54 GB took 4 hours and a half.
Besides Borg and Arq, I tested again Duplicacy, Kopia and Restic to see if they have changed since last time I checked them. I have used both Duplicacy and Restic in the past, and only tried Kopia once, but didn't want to use it because people said it was not mature and dependable yet.
With Duplicacy, I like the GUI etc but even if people say it's reliable, I got a nasty surprise yesterday. My backup was interrupted at some point due to a disconnection for which I had to restart my router. I resumed/restarted the backup with no errors etc I could see in the UI, but when I tried to test a restore, the snapshot would not list any files at all. I researched a bit and the only solution was to delete all the snapshots to be able to back up again and reuse the existing chunks. But basically if I had been in the need to restore from that backup, I would have been screwed. So I passed again on Duplicacy.
Kopia is super fast, but I cannot ignore the reports of repository corruption etc still quite recently (I actually had that kind of problem myself when I tried it the first time and I was hoping these issues would have been fixed by now). So I passed on Kopia as well.
Then I tried the latest version of Restic, but this time with a couple of new settings that I discovered today: RESTIC_READ_CONCURRENCY and s3.connections. I have NVME drives, and use Storj, so I set both to 100. I was shocked! Like with Kopia, backups max out my Gigabit connection using Storj, but the biggest surprise is that it also maxed out the speed when restoring, while Kopia was using less than half of my bandwith. So with Restic I got both backups and restores at around 120-122 MB/sec which is absolutely awesome!
Restoring the same 54 GB of data took 4.5 hours with Arq/Storj, 43 minutes with Borg/NetDynamics, and only 7 minutes with Restic/Storj!! I still can't believe what a difference those settings can make.
I am still going to do some more tests, but it looks like I may switch back to Restic with these new settings, instead of Arq.
What is your current backup strategy? What tools and providers you use and what made you choose them?
Comments
Nice thread, I still need to figure out a strategy...
Do you use any backup tools at the moment?
I have got 10TB local drive plugged into the local network where I save data with quick, local access. I have also got 2TB backup server in NL where I installed restic with webui where I backup some, more precious data. I am also using storj for limited backups. I also use Amazon Glacier for cheap storage of heavy data like videos and usual S3 to backup DBs, invoices, etc. I am also using imap-backup to backup emails three times a day (like 3am, 3pm, and 8pm).
P.S. my pics from my phone at being syncthinged to my local raspberri pi machine, where they are being archived and backed up to storj.
I backup to GitHub 😶🌫️
Restic, Kopia, Zstd/zip + GnuPG.
I have three types of information. Files, websites and management applications.
I have all the files in the cloud. I synchronize those files with other providers with rsync in different periods, always leaving those backup servers disconnected.
I make a weekly copy of each website.
Finally the application with database, I make a full backup of the database daily within the same provider (other storage) and once a week to two different providers.
Once a year I also download a copy of everything that I leave offline.
Is there any decent GUI for restic yet ?
I use Promox backup server to backup / restore my VMs
For the rest of my data I have remote backups on storage VPS with borg.
I have 2 storage VPS in different locations. It's not expensive and it works well.
I don't back up my laptop because everything important ends up on a NAS which is backed up with borg.
I'll try Kopia. 1st time I heard of it. Thanks
Which settings do you use with restic?
Why both Restic and Kopia?
So do you manage snapshots manually with rsync>
I tried Backrest and it wasn't particularly good, so I am just using the CLI
Which provider do you use for the storage VPS?
Hosthatch (RAID 1) / HostC (RAID 6)
Both have the same copy of the data
They are also used as a proxmox backup server.
So I don't have to pay for another service for this use.
Cheap setup given the storage amount. It's simple and I feel safe.
Kopia for personal computer, Restic for servers.
Use JetBackup on the cPanel servers, I use PBS for VMs, and Veeam for VMs and physical servers. I also rsync some of the data on the SANs that PBS and Veeam use between DCs and to a cloud provider.
I don't do backups atm
I really need to get a nice backup solution going with a staging area for testing.
All my servers have the same backup setup, Restic+Kopia in docker container & monitoring though Uptime Kuma.
I run two different software because I don't want risk any data lose in repository corruption. There was a time when I used to back up one by one application in archive format then transferred (Transfer happened in a separate server) them two different storage providers. If there is any problem on any of those backup, I might just lose one application data for a single backup point. Now with restic & kopia's repository architect, the risk is high.
I use hosthatch storage VPS with garage S3 setup & used iDrive E2 which I recently removed as they increased my 1 TB plan price from $15 a year to around $50. I might shop for a storage server on this Black Friday or just store them on Google Drive(iDrive replacement for now).
Borg
How has the uptime been with these? A colleague needs some storage and I suggested the Hetzner Storage Box for simplicity but performance isn't great.
Aren't you worried about the reports about repository corruption with Kopia?
By PBS do you mean Proxmox?
Ouch.... nothing at all?
Kopia is nice and fast but I don't feel like trusting it yet from what I have read.
And for the storage?
Restic backup 1 to local external drive
Restic backup 2 to remote cloud drive 1
Restic backup 3 to remote cloud drive 2 if backup 1 wasn't possible for whatever reason
Occasionally try restoring those backups (partially, if space too small) on an idling VPS, another type of "benchmark" while verifying you backup is still working
With restic it’s nice that you can tell the check command to verify the integrity of the backups by restoring a subset of the data. Pretty handy
Nextcloud for backups / synchronization between devices (laptops / PC and phone) of important data (200 GB, but I only really need my KeePass DB) on one self hosted Proxmox VM
Proxmox Backup Server for backing up server side stuff (again nextcloud, but also mails, webserver, etc) using two remote PBS servers, with varying intervals
main laptop with work stuff also backed up to one of the mentioned remote servers using borg (vorta backup to be precise). mainly /home but also others, this again includes the most important data of the nextcloud stuff
Phone is backed up using neobackup (for apps, rooted) occasionally and using Nextcloud auto upload of photos. Contacts and calendars are also synced using Nextcloud. I reset / reflash my phone regularly (quarterly to monthly actually)
I migrate a couple of VMs yearly, so I test my VM backups occasionally and partially as well
My really important data is therefore replicated among 3-7 devices using various formats (actual files, proxmox backup, borgbackups)
I do not have a homeserver (costs of hardware & electricity, CGNAT / global availability, and) because I believe that people earning money off of building stable and available systems might offer much more stability/reliability in the long run. Further, bigger systems are cheaper per unit than building said unit yourself imho. My 3TB hosthatch server costs 75€/y with RAID 10 and 20 GB NVMe. A RPi with that (storage) hardware would easily be in the region of 200€+ setup costs, disregarding network connectivity+ costs, performance differences, cooling, power redundancy, ECC and so on. Considering that + that a provider such as hosthatch does actually have spare drives and likely do replace the whole hardware after a couple of years (I assume 5ish), I think I couldn't compete with a homelab that easily for my use case (note, off-site backup is still required in both cases here)
I believe this changes when using better hardware pretty fast however (16TB club or similar), but then also electricity costs rise as well (~ 0,25c/kWh is what I pay => ~15€/100W sustained per month)
I think about maybe storing my Linux ISOs at home however
Looks like you are quite well covered for backups 👍
Several people have mentioned in this and other thread about Hosthatch storage servers. Have they been reliable for you?
BTW I decided to give iDrive e2 another shot as a secondary location, since it's anyway cheaper than Wasabi at 2.5e/TB/mo for the first year instead of $6.99/TB/mo. I have Storj as primary destination anyway.
I bought Koofr package from Stacksocial. As you have used Restic, is this the way to backup using Restic?
https://koofr.eu/blog/posts/backup-your-files-to-koofr-using-restic
Yeah Rclone can be used together with Restic for storage types not natively supported by Restic. It works, but it can be significantly slower that way. Never heard about Koofr before.
Restic and Borg daily backups to dedicated servers in different continents, so that each machine has 2 backups.
I like Restic because you don't need to run anything special on the server, while with Borg you need to install the Borg binary and have it reachable via ssh or other transport. I also like that you can restore to any directory with Restic, as opposed to Borg which restores to the current directory only.
I've tried Duplicati and other Dupli* software before, but backup were very slow and the backup repository kept getting corrupted.
Absolutely, yes! No performance problems, no network hiccups (that I noticed), no nothing
Only problem with hosthatch could be support time, but as I only have one backup server with them the response time is not super duper critical for me (and I believe they suggest getting non promo servers with them if that is the case)
Duplicati is probably the least reliable backup tool ever
Gotcha. Woud you use these servers instead of Hetzner's storage boxes for plain storage of data (not backups)?
Can you please provide me any tutorial link for a faster way to back up?
Just use Restic directly if possible, without Rclone. But the storage must be supported, so it's easier if you use some kind of s3 compatible object storage service or other type of storage that Restic supports natively. At the moment I am using Storj and iDrive e2. You can see here which storage types you can use directly with Restic.
Duuuuuude! Get on the backup train! 🤦🏻♂️ you cant afford not to backup. Quick back story and how I learned my lesson.
13 or so years ago we had a large number of customers resold through cvps, well, it was right at the time of the big solusvm hack. Cvps got hit hard! Still feel bad for Chris, they lost a lot of customers that day. Welllll, one of my personal vps was on his servers, no backups of my own, didn't really believe I needed them 🤷🏻♂️🤣
That personal vps held my own personal btc wallet and node. At the time, last I checked, had over 1000 bitcoins in that wallet file 😐🙃 needless to say lost em all. That day I learned my lesson good 😂 every so often I check old drives and such for that wallet, not found it yet. Chris is you happen to have one of those old backup drives holding my vpses I'll split it with you!! 🤣
I now backup my stuff religiously. Now my backups are very biased since I own a dc, so I backup to my stuff first haha. Backup straight to our in house backup server at the dc, and also to our offices personal server has a large backup drive. All our staff uses linux for office PCs so just simple rsync once a day. But that should give customers some peace of mind, I use my own stuff for backups of my own.
Then we have a script that runs on the backup server once a day to sync our backup directory to aws glacier cold storage.
All customers data in shared and vps nodes goes to local separate backup drive on each node, then synced to one of our backup server drives.
Backup servers all use zfs raidz2, with daily snapshots. I'm very happy with zfs, it does a great job of self healing partitions when there is an issue.
Simple and effective, hasn't let me down.
My lesson was when I was around 14-15 years old. I lost a lot of source code for games etc I had written because of a virus, and since them I have always been very careful with my backups, and always have redundancy both on site and off site.
People who don't make a remote backup are either crazy or don't have any important data.
If you find yourself in the first situation, you're going to learn the hard way
Pretty much the same story for Bitcoin. Except that I had 6 and not 1000.
Since then, I've taken backups seriously.
Uptime: > 99.99% for both
For hosthatch, in fact, the last outage was months ago and was a network outage of 1m30
And Host-C had scheduled maintenance for a few hours. Apart from that it's very stable.
I was also using Hetzner storage boxes before that, and I was satisfied with the performance.
But now I need VPS to run PBS and the value for money is even better.