All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How do you backup very large data?
Hard drive failures are rare but they do happen. For very large amount of data, normal backup would be pain if using the methods I could think of like rsync, OS image.
I just wonder how do you backup your data if it is very large. Let say hundreds of GB or TB. Do we have efficient way or I can only pay the provider to do the job instead.
I do not have a very large site in question, just want to learn so please don't troll me.
I got the question after reading this on a blog
We are sending you this email as we have discovered that two drives that mirror the same data have failed, which means any data left on the RAID 10 storage array became irrecoverable.
While this is an unfortunate situation we would like to avoid, computer hardware failures do happen and we are not responsible for any lost data. Again, we would like to stress the importance of making backups of important data.
We are now setting up a new VPS for you and you will be able to access it through the original IP (202.5.xx.xx) within the next 24 hours.
We apologise for any incovenience. Please contact us should you have any questions.
Regards,
HostUS
Comments
Just takes a little longer, where's the problem?
Backups also consume server resource, so I just wonder if it takes too long that even makes the server unusable at the time. For a large backup, it may take too long as well.
Nope.
But you could limit the bandwidth. etc.
Backups never take too long.
If its a low spec machine limit cpu, io and network bandwidth accordingly.
Is compress and transfer TBs of data that easy? I do not doubt you but I was never in this case before.
Machines don't feel pain
If your backups aren't automated, then you have to automate them
If automating is a pain for you, pay someone else to automate them for you
Don't run the backup service with too favourable niceness and with (often useless) time-consuming compression methods; prefer incremental backups over constant server imaging
Easy like sunday morning.
Seems like not a nightmare as I think.
Not sure why I have sympathy for machine, afraid it is getting tired for doing large job lol.
Thank you for your help.
LOL.
Don't worry.
If it can't handle backups get another one.
Ain't no sunshine while she's gone.
Take a snapshot of who system is very easy and transfer to backup server/cloud. Example: 350GB VM can take 30 - 40 mins to compress and transfer (depends on network and CPU on your server)
For images / video it only needs to be backed up once, as in the file won’t change over time nf830t.jpg will always be the same then you just need to mirror it across providers and instances. It’s just simply pushing to one server that then pushes to several others and if it exists then skip.
Incremental backups are your solution. Tar can do then very nicely.
I use rsnapshot and it's fast. I've used rsnapshot to backup 400GB on a VPS limited to 100Mhz CPU, 128MB RAM and 10Mbps network pipe. The server didn't break a sweat.
Percona XtraBackup helps a lot with no locking when dumping 500+ gb database. And copy dump to remote server is easy with rsync or any other tool.
Backup complexity comes when you have 2+TB database with millions, even billions entries. I didn't encounter such infrastructure yet.
Another issue with backups is the data being out of date pretty much immediately. Imagine a forum, that get's constant posts to it. 1 hour old backup is already out of date. But of course it's just a forum. Who cares right? But almost everything gets constant changes and backups can be nightmarish if the data is very valuable.
This is why slave DB servers exist.
Use a better compression algorithm with better compression ratios Like xz or zstd. Three of my write ups you might be interested in reading
Read up / search for how to use nice/ionice to control process prority as well i.e. https://www.cyberciti.biz/tips/linux-set-io-scheduling-class-priority.html. It's a must have for any decent backup script/process
Just a note... if using rsync you can pass parameters to nice/ionice the remote end as well.
For databases, use replication.
For static files, use rsync for incremental backup.
I don't see a reason why you need to transfer terabytes of data every day or two.
Thank you guys for pointing it out, I always use incremental snapshot but never know it is also applied to specific data. Where I have been living.
I paid the providers to do the backup for my image server, should do it myself by now.
@msg7086: What are your feeling of always seeing "RAM" in LET.
Good luck with your backup.
Playing Mario while backup.
Sounds reasonable.
I use raspberrypi as a NAS and a primary backup node and I feel no sympathy for this little one working days and nights to deal with daily cronjobs managing tens of terabytes of data. I bet you are less abusive and conscientious person to feel sympathy for your machine running a backup. You don't have to worry about machine's rights until they rebel against us.
Seems like nobody told the obvious answer: it's too big and/or too annoying to backup? Have no backup! This is LET, we all know that BACKUPS ARE FOR THE WEAK, isn't it?
Tetris.