Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Proxmox and backups question
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Proxmox and backups question

Hello. Lets say if my raid who proxmoxs boots of, and all my vms are stored on fails 100%, how will it work to recover the backups stored in a different raid on the same node? Is it just to replace the faulty raid, mount the backup raid, and restore from backups, or will i run into issues here ? Planning a long term host for a few important servers here, but i have never experienced a hardware fault while running proxmox, so looking for some more information.

Comments

  • jvnadrjvnadr Member
    edited April 2019

    What type of raid? Do you have physical access to the server or is it on a remote environment?
    In any case, RAID is NOT a backup system. So, you have to use ALWAYS offline backups and, if it is a critical service, not just one but several.

    In your case, if the server is not booting at all, then, it is a strong possibility that you have lost all of your data or recover will be tricky in case of a hardware RAID. If it is a RAID 1, then, there is some possibility that you can recover data if you have access to the drive. If it is RAID 10, it will be more tricky, it can be possible if only one disk has failed. If you have hardware raid and the raid card messed with the drives, then, it will be extremely tricky to recover data from the array.

    What do you mean "a different RAID"? Do you have more than one RAID array in your server? I think you are confusing disks with RAID in your text. Do you mean this?

    Johnsendrift said: how will it work to recover the backups stored in a different raid disk on the same node? Is it just to replace the faulty raid disk , mount the backup raid, and restore from backups, or will i run into issues here ?

    Or are you planning to have an on-site backup (a backup on the same physical server) using different disks on a second RAID array, mirror the initial setup there and make it live when the first RAID becomes unavailable?
    Wouldn't it be more easy and logical to use for such a setup a second server, mirror there your setup and make the server live if the first one becomes unavailable, using an external DNS service to handle your domain(s)?

    Thanked by 2plumberg amstel
  • Was planning to backup the vms to a secoundary raid in the same server. It is a remote server, no access to it..

  • FalzoFalzo Member

    @Johnsendrift said:
    Was planning to backup the vms to a secoundary raid in the same server. It is a remote server, no access to it..

    please be more informative about what setup you have in mind... if you expect people to advice, they need to see the full picture.

    a 'secondary raid' can be anything ranging from hw-raid (what level) of two additional HDDs to a sw-raid (again, what level?) of two partitions but same disk as the primary one.

    best guess: if you lose your first raid array that includes your OS, you will need to replace it and reinstall your system. you then might be able to reassemble your second raid and restore your vms from that backup. or you might be able to use some kind of rescue mode to get into the box, mount your second raid and move your backups out before reinstalling the whole server.

    TL;DR; as @jvnadr pointed out it most likely is not the best idea to hold your (only) backups on the same server. yes you might be able to restore from that, but I'd consider the risk of something going wrong in that process quite high.

  • Alright, i will try to explain my question a little better :

    My host, 1 single node for this project, running multible vms. The OS, and the vms are running of the same raid volume, of 2 x 1tb nvme`s. I want to make the vms easy to restore if this raid fails.

    So, i want to have a secound raid volume, of 2 x 10TB drives, OR 2 single drives with no raid, as its just for backups. I use proxmox automatic backup feature to backup the vms to one drive, or raid volume (not yet created, i dont know yet)

    So, if my raid with OS and nvme fails, and server cannot boot. Can i replace the drives, install the same version of Debian and Proxmox, mount my backup drive (or raid volume), and simply restore the vms ?

    Only running software raid in Debian. Because of this, im considering just running a drive with no raid as backup. This drive is backed up somewhere else also.

  • launchvpslaunchvps Member, Patron Provider

    It sounds like you have 2 x RAID arrays in the same server? One array for running VMs and another to store backups to?

    If this is the case, yes this will work as you described. But, if you could, use at least RAID 1 the backup drives. If you need to restore data in the future it could be a long process and a drive could fail during the restore process.

    Ideally, you would want to send your backups off the server using an NFS server or something similar. This would allow you to have the data available for any hardware node easily. This would be a great solution if your hardware node failed in some way where it could not boot (ie. motherboard failure). Then, you could stand up any new server and migrate your backed up VMs to it. I suppose you could do it with your RAID array method, but it can be more complicated with the possibility of moving drives, importing configs, differing RAID cards, and firmware.

    Finally, if you use NFS, you can grow your 1 server node into a larger cluster and use the NFS share amongst your entire cluster for backups. You can also store your templates and ISOs here as well. From there, all of your nodes can access those resources as well as store backups there.

    If you decide to grow the cluster in the future, the backups solution you are thinking of using now could get more complicated.

    The NFS server doesn't need to be a beefy server. A simple modest server will do.

    Thanked by 2uptime amstel
  • JohnsendriftJohnsendrift Member
    edited April 2019

    Thank you for your reply. I just got this host and havent set up the backup yet. How is it to import a raid in a new debian installation if i raid the backup space ? I always offload backup files to a file server. But that server is only on a 100mbit connection, so i want to store the backups from this host locally too, as the vms on this server will use alot of space.

    My main question right now will be : Is it overkill to raid backup drives here, when it is already stored offsite ? Drives aint cheap.

  • launchvpslaunchvps Member, Patron Provider
    edited April 2019

    If you are using a software RAID like mdadm, importing shouldn't be too much of an issue as this is supported natively in Linux.

    If you are using hardware RAID, you would need to be sure you are moving the drives to the same hardware RAID controller or a controller that supports importing the RAID configs from the previous controller. Usually, this will be an option for a newer version of the previous RAID controller.

    Once you import the configuration successfully, then you can boot into Debian and the data should be accessible.

    I would definitely be sure there is a local (as in the same datacenter) backup of your data. You specified this will be a lot of data. If you need to restore, you will need this data to be local so you can come back from a failure in a reasonable amount of time. If you do not have local copies and attempt to restore via an offsite backup over a 100mbit connection, this could be very time consuming if you have several TB of data which it sounds like you may have.

    If you can, get an additional server in the same datacenter to be your NFS server. If you can't do that, the next best thing is having that additional RAID 1 for your VM backups. In addition to one of the above, you can send backups offsite. The offsite should really be your last resort if you need to perform a restore - not the first recovery location.

  • FalzoFalzo Member

    @Johnsendrift said:

    My main question right now will be : Is it overkill to raid backup drives here, when it is already stored offsite ? Drives aint cheap.

    it's probably more a matter of how valuable your data is to you ;-) I would not use raid just to not have to reassemble(import) that after a failure. yes mdadm should do that with one line, but what about if the array got split while your system went to hell...

    whatever you decide to use and do, you should test it! which means go the whole way from setting all up, create a vm, make a backup, bork up your system on nvme, reinstall, gain access to your local backup, restore from that backup.

  • @Falzo said:

    @Johnsendrift said:

    My main question right now will be : Is it overkill to raid backup drives here, when it is already stored offsite ? Drives aint cheap.

    it's probably more a matter of how valuable your data is to you ;-) I would not use raid just to not have to reassemble(import) that after a failure. yes mdadm should do that with one line, but what about if the array got split while your system went to hell...

    whatever you decide to use and do, you should test it! which means go the whole way from setting all up, create a vm, make a backup, bork up your system on nvme, reinstall, gain access to your local backup, restore from that backup.

    Going to do that later today and test it. Will post results.

  • Created a crash situation now, worked flawless for me. Thank you for your reply.

    Thanked by 1Falzo
  • launchvpslaunchvps Member, Patron Provider

    @Johnsendrift said:
    Created a crash situation now, worked flawless for me. Thank you for your reply.

    Good to hear. :) Hope it works out for you.

  • jvnadrjvnadr Member
    edited April 2019

    A second raid is much more expensive than a couple or even more cheapo storage vps that can be your backups. Get 2 vps (@Anthonysmith for example do have some excellent and cheap storage vps for that, even NAT can work) and use them instead of a second raid in your server. You can mirror your infrastructure to the first of the vps and do a cold or zipped backup on the second, to be sure in any case... This would be a primary secure setup, not to lose your precious data.
    Also, the RAID in your server? A RAID 1 would be good for some safety if one single disk fails, RAID 0 to speed up the disks especially if they are not SSD. But, again, this is not any kind of backup. If you do not have any offsite backups, then, you do not have any backups at all. And in my opinion, if you have only a single backup, again, it is likely you don't have backups at all.

    Johnsendrift said: My main question right now will be : Is it overkill to raid backup drives here, when it is already stored offsite ? Drives aint cheap.

    Yes. Two or three backup vps (not powerful, just for backing up things) would be cheaper and far better solution.

  • @jvnadr said:
    A second raid is much more expensive than a couple or even more cheapo storage vps that can be your backups. Get 2 vps (@Anthonysmith for example do have some excellent and cheap storage vps for that, even NAT can work) and use them instead of a second raid in your server. You can mirror your infrastructure to the first of the vps and do a cold or zipped backup on the second, to be sure in any case... This would be a primary secure setup, not to lose your precious data.
    Also, the RAID in your server? A RAID 1 would be good for some safety if one single disk fails, RAID 0 to speed up the disks especially if they are not SSD. But, again, this is not any kind of backup. If you do not have any offsite backups, then, you do not have any backups at all. And in my opinion, if you have only a single backup, again, it is likely you don't have backups at all.

    Johnsendrift said: My main question right now will be : Is it overkill to raid backup drives here, when it is already stored offsite ? Drives aint cheap.

    Yes. Two or three backup vps (not powerful, just for backing up things) would be cheaper and far better solution.

    Thank you, but getting vps servers for my files will be even more expensive. There is around 3 TB of data currently running. A 3TB vps is more expensive than a 10TB hdd from Hetzner, and it will eat by bandwidth like popcorn.

    Currently backing up locally (2 saved backups at any given time locally), and offloading to another server in Hetzners network to avoid eating bandwidth

Sign In or Register to comment.