Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How do you backup very large data?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How do you backup very large data?

banxixbanxix Member
edited February 2019 in Help

Hard drive failures are rare but they do happen. For very large amount of data, normal backup would be pain if using the methods I could think of like rsync, OS image.

I just wonder how do you backup your data if it is very large. Let say hundreds of GB or TB. Do we have efficient way or I can only pay the provider to do the job instead.

I do not have a very large site in question, just want to learn so please don't troll me.

I got the question after reading this on a blog

We are sending you this email as we have discovered that two drives that mirror the same data have failed, which means any data left on the RAID 10 storage array became irrecoverable.

While this is an unfortunate situation we would like to avoid, computer hardware failures do happen and we are not responsible for any lost data. Again, we would like to stress the importance of making backups of important data.

We are now setting up a new VPS for you and you will be able to access it through the original IP (202.5.xx.xx) within the next 24 hours.

We apologise for any incovenience. Please contact us should you have any questions.

Regards,
HostUS

Thanked by 1eol
«1

Comments

  • Just takes a little longer, where's the problem?

  • banxixbanxix Member
    edited February 2019

    @eol said:
    Just takes a little longer, where's the problem?

    Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time. For a large backup, it may take too long as well.

  • @banxix said:

    @eol said:
    Just takes a little longer, where's the problem?

    Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time.

    Nope.
    But you could limit the bandwidth. etc.

    @banxix said:
    For a large backup, it may take too long as well.

    Backups never take too long.
    If its a low spec machine limit cpu, io and network bandwidth accordingly.

    Thanked by 2banxix uptime
  • banxixbanxix Member
    edited February 2019

    @eol said:

    @banxix said:

    @eol said:
    Just takes a little longer, where's the problem?

    Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time.

    Nope.
    But you could limit the bandwidth. etc.

    @banxix said:
    For a large backup, it may take too long as well.

    Backups never take too long.
    If its a low spec machine limit cpu, io and network bandwidth accordingly.

    Is compress and transfer TBs of data that easy? I do not doubt you but I was never in this case before.

  • mfsmfs Banned, Member
    edited February 2019

    banxix said: normal backup would be pain if using the methods I could think of like rsync, OS image.

    Machines don't feel pain
    If your backups aren't automated, then you have to automate them
    If automating is a pain for you, pay someone else to automate them for you

    banxix said: if it takes too long that even makes the server unusable at the time.

    Don't run the backup service with too favourable niceness and with (often useless) time-consuming compression methods; prefer incremental backups over constant server imaging

  • @banxix said:

    @eol said:

    @banxix said:

    @eol said:
    Just takes a little longer, where's the problem?

    Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time.

    Nope.
    But you could limit the bandwidth. etc.

    @banxix said:
    For a large backup, it may take too long as well.

    Backups never take too long.
    If its a low spec machine limit cpu, io and network bandwidth accordingly.

    Is compress and transfer TBs of data that easy? I do not doubt you but I was never in this case before.

    Easy like sunday morning.

  • Seems like not a nightmare as I think.
    Not sure why I have sympathy for machine, afraid it is getting tired for doing large job lol.

    Thank you for your help.

    Thanked by 2eol mfs
  • LOL.
    Don't worry.
    If it can't handle backups get another one.

    Thanked by 1banxix
  • @eol said:

    @banxix said:

    @eol said:

    @banxix said:

    @eol said:
    Just takes a little longer, where's the problem?

    Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time.

    Nope.
    But you could limit the bandwidth. etc.

    @banxix said:
    For a large backup, it may take too long as well.

    Backups never take too long.
    If its a low spec machine limit cpu, io and network bandwidth accordingly.

    Is compress and transfer TBs of data that easy? I do not doubt you but I was never in this case before.

    Easy like sunday morning.

    Ain't no sunshine while she's gone.

    Thanked by 1eol
  • WebProjectWebProject Host Rep, Veteran

    Take a snapshot of who system is very easy and transfer to backup server/cloud. Example: 350GB VM can take 30 - 40 mins to compress and transfer (depends on network and CPU on your server)

    Thanked by 1banxix
  • @Letzien said:
    Ain't no sunshine whilen she's gone.

  • For images / video it only needs to be backed up once, as in the file won’t change over time nf830t.jpg will always be the same then you just need to mirror it across providers and instances. It’s just simply pushing to one server that then pushes to several others and if it exists then skip.

    Thanked by 1banxix
  • FHRFHR Member, Host Rep

    Incremental backups are your solution. Tar can do then very nicely.

    Thanked by 1banxix
  • I use rsnapshot and it's fast. I've used rsnapshot to backup 400GB on a VPS limited to 100Mhz CPU, 128MB RAM and 10Mbps network pipe. The server didn't break a sweat.

    Thanked by 1uptime
  • Thanked by 1WebProject
  • Thanked by 1eol
  • LeviLevi Member
    edited February 2019

    Percona XtraBackup helps a lot with no locking when dumping 500+ gb database. And copy dump to remote server is easy with rsync or any other tool.

    Backup complexity comes when you have 2+TB database with millions, even billions entries. I didn't encounter such infrastructure yet.

  • randvegetarandvegeta Member, Host Rep

    Another issue with backups is the data being out of date pretty much immediately. Imagine a forum, that get's constant posts to it. 1 hour old backup is already out of date. But of course it's just a forum. Who cares right? But almost everything gets constant changes and backups can be nightmarish if the data is very valuable.

  • @randvegeta said:
    Another issue with backups is the data being out of date pretty much immediately. Imagine a forum, that get's constant posts to it. 1 hour old backup is already out of date. But of course it's just a forum. Who cares right? But almost everything gets constant changes and backups can be nightmarish if the data is very valuable.

    This is why slave DB servers exist.

    Thanked by 1vimalware
  • banxix said: I just wonder how do you backup your data if it is very large. Let say hundreds of GB or TB. Do we have efficient way or I can only pay the provider to do the job instead.

    Use a better compression algorithm with better compression ratios ;) Like xz or zstd. Three of my write ups you might be interested in reading

    Thanked by 2banxix plumberg
  • banxix said: Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time. For a large backup, it may take too long as well.

    Read up / search for how to use nice/ionice to control process prority as well i.e. https://www.cyberciti.biz/tips/linux-set-io-scheduling-class-priority.html. It's a must have for any decent backup script/process :)

    Thanked by 1banxix
  • eva2000 said: Read up / search for how to use nice/ionice to control process prority as well

    Just a note... if using rsync you can pass parameters to nice/ionice the remote end as well.

  • For databases, use replication.

    For static files, use rsync for incremental backup.

    I don't see a reason why you need to transfer terabytes of data every day or two.

    Thanked by 2randvegeta banxix
  • banxixbanxix Member
    edited February 2019

    Thank you guys for pointing it out, I always use incremental snapshot but never know it is also applied to specific data. Where I have been living.

    I paid the providers to do the backup for my image server, should do it myself by now.

    @msg7086: What are your feeling of always seeing "RAM" in LET.

    Thanked by 1eol
  • Good luck with your backup.

    Thanked by 1banxix
  • @eol said:
    Good luck with your backup.

    Playing Mario while backup.

    Thanked by 1eol
  • Sounds reasonable.

  • I use raspberrypi as a NAS and a primary backup node and I feel no sympathy for this little one working days and nights to deal with daily cronjobs managing tens of terabytes of data. I bet you are less abusive and conscientious person to feel sympathy for your machine running a backup. You don't have to worry about machine's rights until they rebel against us.

    Thanked by 2banxix uptime
  • Seems like nobody told the obvious answer: it's too big and/or too annoying to backup? Have no backup! This is LET, we all know that BACKUPS ARE FOR THE WEAK, isn't it?

    Thanked by 2eol banxix
  • Tetris.

    Thanked by 1eol
Sign In or Register to comment.