Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Critical Data Backup - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Critical Data Backup

2

Comments

  • @user123 said:

    @TimboJones said:

    @Daniel15 said:

    @yoursunny said: If data is corrupted, rclone would faithfully synchronize the corrupted files to the backup.
    Oops.

    On Linux systems I use Borgbackup since it lets me keep months of backups relatively cheaply (as it dedupes blocks) and I can go back to an uncorrupted file. I still haven't quite figured out what to use for backups on my Windows systems. My approach right now is to throw all the important files into Seafile, which is hosted on a HostHatch VPS, which is then backed up to a Servarica VPS.

    Veeam agent. Free. Incremental updates. Restore to different hardware, etc.

    I like Veeam Agent because it's easy to use (disclosure: I've never tried to restore a backup from it), but the biggest downside I run into is that you can only run one job with the free version of Veeam Agent. I think the free version of Veeam Backup & Replication lets you have up to 10 jobs, but it appears to use more resources and focused on backing up VMs. I'm only backing up physical computers, not VMs.

    Since my files aren't very well organized and Agent only lets me set up one backup job, I have to do a complete drive backup. This requires much more space than would be needed if I could exclude certain folders (e.g. those containing media files, temporary files, etc.), which also limits the number of copies of previous backups I can keep.

    What? No. You can do incremental and periodic full backups and delete old versions with free on a daily backup job.

  • TimboJonesTimboJones Member
    edited December 2021

    @user123 said:
    Going back to my point about Veeam Agent's limitations, it seems like a very strange decision to allow individual workstations to support multiple backup jobs ONLY when configured to do so by an external (VBR) server and only one backup job if configured locally without an external server.

    This is common for free vs paid limited options. Basic vs advanced/power user.

  • @TimboJones said:

    @user123 said:

    @TimboJones said:

    @Daniel15 said:

    @yoursunny said: If data is corrupted, rclone would faithfully synchronize the corrupted files to the backup.
    Oops.

    On Linux systems I use Borgbackup since it lets me keep months of backups relatively cheaply (as it dedupes blocks) and I can go back to an uncorrupted file. I still haven't quite figured out what to use for backups on my Windows systems. My approach right now is to throw all the important files into Seafile, which is hosted on a HostHatch VPS, which is then backed up to a Servarica VPS.

    Veeam agent. Free. Incremental updates. Restore to different hardware, etc.

    I like Veeam Agent because it's easy to use (disclosure: I've never tried to restore a backup from it), but the biggest downside I run into is that you can only run one job with the free version of Veeam Agent. I think the free version of Veeam Backup & Replication lets you have up to 10 jobs, but it appears to use more resources and focused on backing up VMs. I'm only backing up physical computers, not VMs.

    Since my files aren't very well organized and Agent only lets me set up one backup job, I have to do a complete drive backup. This requires much more space than would be needed if I could exclude certain folders (e.g. those containing media files, temporary files, etc.), which also limits the number of copies of previous backups I can keep.

    What? No. You can do incremental and periodic full backups and delete old versions with free on a daily backup job.

    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

  • @TimboJones said:

    @user123 said:
    Going back to my point about Veeam Agent's limitations, it seems like a very strange decision to allow individual workstations to support multiple backup jobs ONLY when configured to do so by an external (VBR) server and only one backup job if configured locally without an external server.

    This is common for free vs paid limited options. Basic vs advanced/power user.

    I agree with you...except that as @aj_potc has pointed out, configuring Agent through the free version of B&R suddenly allows you to configure Agent to run multiple jobs.

  • TimboJonesTimboJones Member
    edited December 2021

    @user123 said:
    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

    Because it's incremental AND you can do file level restore from image backup, there's less point for two image and file backup jobs for simple use cases.

    @jbiloh just had a case of comment posting greyed out. Refreshed page and latest content I wrote wasn't displayed but did allow it to post.

  • I have been using a External WD My Passport Portable Hard Drive, for a long time, it work great, it come with Encryption software., it just plug to my PC and it automatically backup my files.

  • user123user123 Member
    edited December 2021

    @TimboJones said:

    @user123 said:
    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

    Because it's incremental AND you can do file level restore from image backup, there's less point for two image and file backup jobs for simple use cases.

    I know you can do file level restore. But, image backup requires much more space and there are multiple backup storage locations. So, it would be nice if I could store multiple versions of the more important folders locally on a NAS and keep multiple versions of the 10-20x larger backup that contains less important data to a storage VPS (keeping a copy of just the one most recent version locally on a NAS). If I ended up wanting to do an image level restore, I'd have that option, but I would have more copies going back further of the important files - i.e. in case something was corrupted.

  • What to do if my critical data is my 8TB movie collection?

  • @Shot2 said:
    For home use, about 500GB of infrequently updated critical data: one on-site backup (scheduled, block-based checksummed disk replication: a kind of asynchronous RAID if you prefer). Convenient for quickly resuming work if the main disk dies: just plug the replica and keep working while waiting for some replacement hardware.

    Plus one "on-demand" (usually weekly) backup to a distant storage server (far away + encrypted + versioned thanks to nilfs2 goodness, da time machine in da cloud). You can bomb my home and my whole country, photos of my fluffy kitty will survive me HAHAHA.

    And what if I make the earth explode ? Do you have backups on the moon ?

  • @Daniel15

    Network backups are only used as a last resort, and I'm backing up at two different providers, in different locations, so hopefully I won't be able to use them.

    @RedSox

    We all have different definitions of what is critical, I basically keep pictures of my kids, the code I write, etc.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @RedSox said:
    What to do if my critical data is my 8TB movie collection?

    Upload to Deep Atlantic Storage, unlimited storage for free.

  • RedSoxRedSox Member
    edited December 2021

    @yoursunny said: Upload to Deep Atlantic Storage, unlimited storage for free.

    Looks good. I uploaded 1TB with no problem. Thank you.

    Thanked by 1yoursunny
  • @user123 said:

    @TimboJones said:

    @user123 said:
    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

    Because it's incremental AND you can do file level restore from image backup, there's less point for two image and file backup jobs for simple use cases.

    I know you can do file level restore. But, image backup requires much more space and there are multiple backup storage locations. So, it would be nice if I could store multiple versions of the more important folders locally on a NAS and keep multiple versions of the 10-20x larger backup that contains less important data to a storage VPS (keeping a copy of just the one most recent version locally on a NAS). If I ended up wanting to do an image level restore, I'd have that option, but I would have more copies going back further of the important files - i.e. in case something was corrupted.

    My point was if you need the features you want, Veeam thinks you should pay. They can't give away the store for free. Adding features to server versions makes sense to get actual business people with servers and budgets to evaluate and get hooked on their software and spend the money.

  • @mcgree said: the code I write, etc.

    I solved this by open sourcing a lot of my code, so multiple people have a copy of the repo :p

    Thanked by 1yoursunny
  • @TimboJones said:

    @user123 said:

    @TimboJones said:

    @user123 said:
    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

    Because it's incremental AND you can do file level restore from image backup, there's less point for two image and file backup jobs for simple use cases.

    I know you can do file level restore. But, image backup requires much more space and there are multiple backup storage locations. So, it would be nice if I could store multiple versions of the more important folders locally on a NAS and keep multiple versions of the 10-20x larger backup that contains less important data to a storage VPS (keeping a copy of just the one most recent version locally on a NAS). If I ended up wanting to do an image level restore, I'd have that option, but I would have more copies going back further of the important files - i.e. in case something was corrupted.

    My point was if you need the features you want, Veeam thinks you should pay. They can't give away the store for free. Adding features to server versions makes sense to get actual business people with servers and budgets to evaluate and get hooked on their software and spend the money.

    My point was that they do give the features for free, if you also install B&R. Moreover, as far as I can tell, they don't even offer the option of buying a standalone version of Agent that supports multiple jobs.

  • @user123 said:

    @TimboJones said:

    @user123 said:

    @TimboJones said:

    @user123 said:
    By "job," I mean you can only select complete drive or volume, or specific files. You can't set Agent up alone to run both a complete drive backup and a file-level backup of specific folders. It's one or the other.

    Because it's incremental AND you can do file level restore from image backup, there's less point for two image and file backup jobs for simple use cases.

    I know you can do file level restore. But, image backup requires much more space and there are multiple backup storage locations. So, it would be nice if I could store multiple versions of the more important folders locally on a NAS and keep multiple versions of the 10-20x larger backup that contains less important data to a storage VPS (keeping a copy of just the one most recent version locally on a NAS). If I ended up wanting to do an image level restore, I'd have that option, but I would have more copies going back further of the important files - i.e. in case something was corrupted.

    My point was if you need the features you want, Veeam thinks you should pay. They can't give away the store for free. Adding features to server versions makes sense to get actual business people with servers and budgets to evaluate and get hooked on their software and spend the money.

    My point was that they do give the features for free, if you also install B&R. Moreover, as far as I can tell, they don't even offer the option of buying a standalone version of Agent that supports multiple jobs.

    https://www.veeam.com/windows-cloud-server-backup-agent.html

    Looks like server version does, workstation only allows unlimited jobs for Veeam Connect Cloud repo. (?)

    Their bread and butter is enterprises, not the personal users.

  • @user123 said:
    My point was that they do give the features for free, if you also install B&R. Moreover, as far as I can tell, they don't even offer the option of buying a standalone version of Agent that supports multiple jobs.

    That's correct. The standalone agents are not something you buy separately.

    Thanked by 1TimboJones
  • @Shot2 said: photos of my fluffy kitty will survive me

    I demand a floof proof my good sir.

  • @TODO said:

    @Shot2 said: photos of my fluffy kitty will survive me

    I demand a floof proof my good sir.

    Here you go https://i.imgur.com/K6kbUFl.jpg

  • is P2P backup software (i backup to your HD, you backup to my hd)
    still around?

  • tholethole Member
    edited December 2021

    @TimboJones said: By that argument, there's no such thing as backups.

    >
    he was talking about "anitya" concept & it's just a matter of time as many don't know any trying to even find a way to backup stuff to human DNA lol

  • jsgjsg Member, Resident Benchmarker

    @Daniel15 already did a good job, but as the issue Raid vs. backup seems to be not clear to some:

    Raid addresses the problem of disks failing, i.e. the medium onto which one puts data.
    Backup addresses the problem of data failing/corrupted/lost/etc for a variety of reasons.

    Finally multiple backups addresses the problem of a backup not being available for a variety of reasons (e.g. house burnt down). Kind of closely related multiple backups on multiple types of media and/or at different locations addresses diverse disaster scenarios, e.g. (electro)magnetics based record being destroyed by e.g. EMP but optical based records not.

  • @thole said:

    @TimboJones said: By that argument, there's no such thing as backups.

    >
    he was talking about "anitya" concept & it's just a matter of time as many don't know any trying to even find a way to backup stuff to human DNA lol

    1. Please don't ever misquote me like that, or anyone for that matter.
    2. I have no clue wtf you're talking about and it's relevance to me or this discussion.
  • Much of this has already been said but here is what I learned.

    1. Any data you care about backup yourself. (3rd parties should be considered as a last resort in the event your own backup fails)
    2. Test your backups. Even try to utilize the backed up data from time to time as an easier way of integrating testing into your workflow. This will help solve the issue with corruption.
    3. Buy new drives every so often. Remember that drives have expiry dates, I have seen 9 year old hard drives in production fail because the person managing them couldn't be bothered to read the manufacturers use by date and order a replacement.
    4. Raid can cause problems if the controller fails and you can't rebuild it so a copy external to the raided version is great. Rebuilding raid can also take days so bear that in mind.
    5. Implement versioning where possible, sometimes the file name or date is a good enough indicator of which version is older
    6. Have multiple copies
    7. Have copies in multiple locations. ( I am sometimes lazy about this ). Also remember to test the backups in the remote location. If you don't test it you can't be sure it will work when you need it.
    8. Make sure you can find the password to the backup when you need it. If a team is involved there should be clear documentation on where the backups are and what programs and password/keys are needed to access it.
    9. Always have an offline backup to protect against ransom wear. Tell them to fuck off as you reinstall and restore from a backup.
    10. Work a reinstall every so often into your workflow. In the event you have a ransom wear attack recovery will be much easier. Have a list of all the applications you use and the configuration files for such. This will also include configurations for network and other IoT devices.

    Depending on how import your data is and what are the consequences of losing it you can implement as much or as little of this as you want.

  • I store my backups to
    1. aws (us-east)
    2. backblaze (us-west)
    3. ovh (sg)
    4. buyvm (ny,lv,lux)
    5. letbox (ny,la)
    6. - still waiting for Elon to launch spacestorageX

  • yoursunnyyoursunny Member, IPv6 Advocate

    @trycatchthis said:
    Buy new drives every so often. Remember that drives have expiry dates, I have seen 9 year old hard drives in production fail because the person managing them couldn't be bothered to read the manufacturers use by date and order a replacement.

    Does it matter whether the drive is active or offline?
    I have two HDD, one is connected to the home router and powered on all the time, the other is normally disconnected and only plugged in occasionally to take a backup.
    Does the offline drive last more years than the active drive?

  • darkimmortaldarkimmortal Member
    edited December 2021

    @jsg said:
    @Daniel15 already did a good job, but as the issue Raid vs. backup seems to be not clear to some:

    Raid addresses the problem of disks failing, i.e. the medium onto which one puts data.
    Backup addresses the problem of data failing/corrupted/lost/etc for a variety of reasons.

    Another often-overlooked thing that RAID addresses is corruption.

    Backups require manual effort discover and restore files affected by corruption (due to bitrot/URE/etc), and you're screwed if the last good copy of the file falls out of retention

    With any good RAID implementation, this is all handled transparently and automatically

  • @yoursunny said:

    @trycatchthis said:
    Buy new drives every so often. Remember that drives have expiry dates, I have seen 9 year old hard drives in production fail because the person managing them couldn't be bothered to read the manufacturers use by date and order a replacement.

    Does it matter whether the drive is active or offline?
    I have two HDD, one is connected to the home router and powered on all the time, the other is normally disconnected and only plugged in occasionally to take a backup.
    Does the offline drive last more years than the active drive?

    Probably will last longer in practice, but still look at the recommended replacement date from the manufacture. I had offline drives fail on me around the 10 year mark.

    Thanked by 1yoursunny
  • jsgjsg Member, Resident Benchmarker

    @darkimmortal said:

    @jsg said:
    @Daniel15 already did a good job, but as the issue Raid vs. backup seems to be not clear to some:

    Raid addresses the problem of disks failing, i.e. the medium onto which one puts data.
    Backup addresses the problem of data failing/corrupted/lost/etc for a variety of reasons.

    Another often-overlooked thing that RAID addresses is corruption.

    Backups require manual effort discover and restore files affected by corruption (due to bitrot/URE/etc), and you're screwed if the last good copy of the file falls out of retention

    With any good RAID implementation, this is all handled transparently and automatically

    Yes and no. Raid doesn't per se, but yes, sometimes Raid does use erasure coding. Note though that checksums usually can detect but not repair data corruption.

    As for corrupted data/files one simple strategy that I found to serve me well for many years is Big-Medium-[Small] cycles like e.g. a Full Backup every X days plus incremental backups every Y hours plus possibly near real time backup logs of volatile data like e.g. new or updated DB records, and an accumulation+deletion mechanism which brings S into M and deletes S after a given number of cycles.

  • darkimmortaldarkimmortal Member
    edited December 2021

    @jsg said:

    @darkimmortal said:

    @jsg said:
    @Daniel15 already did a good job, but as the issue Raid vs. backup seems to be not clear to some:

    Raid addresses the problem of disks failing, i.e. the medium onto which one puts data.
    Backup addresses the problem of data failing/corrupted/lost/etc for a variety of reasons.

    Another often-overlooked thing that RAID addresses is corruption.

    Backups require manual effort discover and restore files affected by corruption (due to bitrot/URE/etc), and you're screwed if the last good copy of the file falls out of retention

    With any good RAID implementation, this is all handled transparently and automatically

    Yes and no. Raid doesn't per se, but yes, sometimes Raid does use erasure coding. Note though that checksums usually can detect but not repair data corruption.

    I mean the mechanism of any RAID where it responds to URE's by reading from another disk and using that to re-write the corrupt data

    Without RAID, when you hit a URE and if you notice it, you are left with a filesystem-specific manual faff to identify the file affected and restore it from backup

Sign In or Register to comment.