New on LowEndTalk? Please Register and read our Community Rules.
Best software: backup
I'm looking for software (preferably open source) to manage backups.
I have quite a bit of data on popular cloud hosts that already have features to meet this need. And I have some automations to export this data to external object storage solutions.
But I'm starting to have quite a few other, less crucial databases on other hosts that I'd also like to back up.
Are there any solutions that can meet this need? With notifications, for example, when a backup couldn't be made?
In concrete terms, do you use any solutions other than manual scripting to handle all your backups?
Comments
What operating system are you using?
~ SMARTHOST
Rclone. It does not have built in notifications or scheduled backup but these can easily be added using simple scripts and cronjobs. Open source.
Generally, debian (latest stable) but some servers are on Ubuntu LTS. This is still debian-based, so it's fairly uniform.
I was thinking I could just back up my databases (often posgresql, sometimes MySQL).
Rather than making disk backups
For critical file storage, I generally use object storage solutions with triple replication.
For the rest, I use docker containers that have no persistent storage on the machine.
I've found some very good solutions. But they're often specialized on a single database system.
I'd like software that can handle all my needs.
I don't know if it's the right approach to back up only the databases. If I settle for that, I'll still lose configuration files. But it would still allow me to get the services up and running again, just with a little more effort.
If you have any advice, I'd love to hear it!
To be honest, rclone is far from being a turnkey backup system.
It requires a lot of scripting, if only to have versioned backups.
I much prefer borg backup / restic (who probably use rclone internally).
But I was looking for solutions even simpler than that. Even with a web GUI.
But I'm not sure what I'm looking for exists.
Proxmox backup server has clients, and there's urbackup as well I believe.
Interesting, but I guess you need access to the host node to use promox backup server?
Making complete snapshots of the VM seems the simplest solution. But I'm not sure it's possible to do this kind of thing with low-cost providers.
Urbackup seems to fit my needs. Not as simple as the solution I have in mind.
But seems flexible enough.
Thanks a lot!
You could try borgmatic. Its simpler to set up than Borg and has integrations with databases.
Indeed, it looks like a good match.
I knew of it by name but had never dug into it.
The only thing missing is a web interface. But there is a desktop client. https://github.com/borgbase/vorta
I don't think I could find a better solution.
Check out the below for server side
https://github.com/Ravinou/borgwarehouse
restic!
Restic...
You could try https://www.duplicati.com/ or https://kopia.io/
I have no personal experience with either, but it sounds like they might fit you
Personally I use restic which works very well, but it does require some scripting.
Aside from the "usual suspects" such as rclone, borg, duplicity,...
Since you are on debian....I also want to throw in "proxmox-backup-client". It works both on a hypervisor level, as well as inside of vm's. Setup may take you a bit, but once its up and running, then its fairly nice.
Since this connects to a "proxmox backup server", you get stuff like a bit of de-duplication, data compression (for storage), server-side data verification, the ability to automatically only retain a certain amount of data (ie: 1 per day for last seven days, 1 per week, one per month,...) depending on your preferences.
Its also easy to sync from one backup "pbs node" to a few more if you want to store your backups in multiple places.
Docs for the client: https://pbs.proxmox.com/docs/backup-client.html
Docs for the server: https://pbs.proxmox.com/docs/
So it would be possible, in theory, to take a snapshot of a complete machine and restore it on just about any provider?
If so, it's well worth the setup time.
I don't see how it could work without access to the hypervisor for both backup and restore, but I'll find out.
Thanks I'll definitely dig into that.
If run on the hypervisor, you get vm (disk) level backups. So if you are working with dedicated servers and run your own hypervisors, then this is something that can potentially be done.
If run inside of a vm, you get "file" backups from inside the vm.
A full automatic "restore" to another provider typically isnt in the cards, because you will need to adjust some config files for the new updated IP-addressing and so on.
However if you can live with that, then just make sure you include the important dirs such as any data directories, /etc for configuration,... For your DB servers i would suggest having it export the databases via sqldump onto a dir on the local file system(so its included in a backup).
Okay, that's clearer
So it's not perfect VM-wise. Because it can be more or less complicated to launch a machine from this backup.
I can think of quite a few cases where it wouldn't be so easy.
But with control over the hypervisor, it's the perfect solution.
I'm going to do a few tests anyway. Thanks
Try UrBackup.
I recommend Restic as it's very easy to use and battle tested. Someone above recommended Duplicati and Kopia.... two of the least reliable backup programs out there
(Do your research)
Yes I know restic. Very reliable
I'll try borgmatic + borgWarehouse.
I haven't had time to get into it yet, but it looks promising
Borg is pretty solid too, but Restic supports lots of backends.
I have been testing PBS for past few days. I wish they had a "restore" option from the PBS. But you have to run it from the client side.
You can not run restore from PBS interface.
Also to setup a client on OS other than Debian/Ubuntu is a PIA. Their are few only docs available but none of them have details on installing latest agent on AlmaLinux 8/9.
It makes a lot more sense to use it with PVE. But to backup for other setups (without PVE) its kind of a pain at this point.
Oooh, I've been using borgbackup directly for a couple of years, but this looks nice. Only problem I can see is that it doesn't support different passphrases per repo, so it looks like I'll have to do some code hacking before I could migrate to this.
cat /dev/sda >> /dev/sdb
ezpz my guy!
I don't understand why would you want to restore from PBS as that would be a huge security risk.
Honestly, I don't think PBS is even supposed to run directly on any host since PVE has more control over firewall.
What I do is PVE on my storage node then PBS inside a LXC. That way I have much better control.
Most service I need are in containers as well as data so I could just add PBS to PVE then hit restore. PBS absolutely make backup and moving between providers painless.
Building for alma/... isnt hard. There are even some "build scripts" for published on github https://github.com/TomGem/proxmox-backup-client (i have not verfied/tesed the content of this repo, as i rund my own build).
I've been looking at duplicacy, seems decent.
You mean having different repos (and their associated passphrases)?
I use borgmatic and have simple shell script wrapper(s) that set up environment variables (BORG_REPO, BORG_PASSCOMMAND, BORG_PASSPHRASE etc.) so that the borgmatic configuration can be reused across different repos with a simple wrapper.
It is very convenient (esp. the hooks based on error conditions) and being able to reuse config files/snippets for (standard) retention across multiple repos.
With a little customization, it's easier to manage (rather than having embedded borg commands). Of course like with all abstractions, you win some and you loose some. So there will be times when it's easier/faster to just go the straight borg route.
I am using https://github.com/restic/rest-server docker image to have central backup server and push nightly backup from VMs.
Had you try it?
What's your last solution?
Yes, I've tried it and it's what I'm using now.
Borgmatic + borgwarehouse. It's great, it works very well.
But it's not quite what I had in mind at first.
It's still not that quick to set up, quite a lot of configuration in config files. In the end, it's not that different from restic.
In my opinion, there's still room for a simpler solution on the backup solutions market.
I also tried promox (VE + backup) with LXC containers (no need for virtualization enabled)
It's just perfect for my use, allowing me to make full server snapshots and easily move /restart vm to another provider.
But it does drain a minimum of system resources. It's not necessarily adapted to all the servers I have, which don't all have a lot of RAM.
I wonder if I'll end up consolidating my servers into one or two larger ones. With promox on each. And save the complete snapshots at home and on a remote storage server.