Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


DIY personal backup solution
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

DIY personal backup solution

alfredalfred Member, Host Rep
edited January 2017 in General

This is going to be quite a long post, but it should be interesting!

I've been thinking about doing remote backups of my home computer with bup.

At first, I used Google's Compute Engine, since their persistent disk storage was cheap and features seemed a bit better than AWS, especially with regard to performance. However, with the VM at $4/mo (and bad specs/perf), and storage at $0.04/GB (also quite bad perf), it didn't really seem worth it.

OK, next try. A small SpeedyKVM VPS in Seattle (which I already had), connected to S3 with s3backer (which basically turns and S3 bucket into a mountable block device). VPS is still $4/mo but performance (CPU-wise) is better, in addition to it being closer to home (Vancouver). s3backer is actually faster than I expected and works well with bup. One advantage of persistence in S3 directly (vs EBS or Google Persistent Disk) is that it's much much more reliable; however, using s3backer reduces this reliability by a bit (because of caching, etc). So I would say durability would be around the same, maybe a bit worse than EBS with snapshots. The real benefit here is cost: S3 is half the price of EBS per GB. Even with the added costs of bandwidth and requests, it should come out to be a bit cheaper. Another advantage is that it has basically unlimited scalability.

That solution is pretty good, but it's still far from perfect. What I'm looking for is:

  • Very high durability
  • Low cost (around the same cost of 2nd solution above: $3-5/mo fixed costs, then $0.02-0.03 per GB; I'm looking for around 500GB - 1TB; scalability is a bonus)
  • Good performance (ideally multiple 10's of MB/s)
  • Good network performance (especially from home; should be located in US west)

To elaborate on durability, above: something like the EBS + snapshot is acceptable. While EBS is not extremely durable, snapshots stored in S3 are, so this is OK.

All things considered, I was thinking of something like a storage VPS, with periodic snapshots (incremental?) uploaded to Google Drive (I have Apps unlimited, so unlimited storage there). My worry is that from what I've seen, decent storage VPS's are quite expensive for the storage, since there are definitely economies of scale in terms of storage.

Thoughts? Ideas?

Comments

  • I am not sure what your end result is trying to be but if you want to do DIY you can use: https://www.cloudberrylab.com/backup/linux.aspx

    Or even just go with BackBlaze B2 Bup and S3proxy. B2 storage is much cheaper.

    Thanked by 1alfred
  • Do you really need s3-style block storage and redundancy? If you just want a cheap storage VPS try @i83 NAT storage (i-83.com) at Beauharnois Quebec (OVH). OVH network is pretty good in US despite being in eastern Canada. Cost should be under 0.01/GB. CPU and iops aren't great but it's fine for basic backup storage. I have one and it's nice.

    @tragic has a 500GB Los Angeles KVM here at $50/year (around 0.8 cents/GB/mo) that I've been tempted by:

    https://www.tragicservers.com/cart.php?a=confproduct&i=0

    It sounds good but I haven't tried it so no experience to report.

    serverhub.com has Openvz storage vps at 0.01/GB (various sizes) with a Seattle WA location. There's a thread about some customers hitting some bad probs with them, but they showed up to fix things, so I still have slight reservations but would probably give them a try.

    OpenVZ.io (INIZ brand) has a Dallas TX site and a good reputation though I've never tried. 0.8 cents/GB in 500GB plan that was out of stock last time I looked.

    I'm not sure about durability issue if your stuff is ending up in Google Apps anyway. If you want dirt cheap, ultra durable, long term cold storage, try online.net's c14 product. It's on a high-redundancy RAID cluster in France and costs 0.002 euro (about 0.21 US cents) per GB per month for storage, plus 0.01 euro per "operation" which means archiving, unarchiving, and deleting (yes they charge you money to delete your data). So it's best for archiving or disaster recovery, i.e. upload once, probably never access, and never delete. Restoring can take hours, like Amazon Glacier.

    Online.net also has a high-availability NAS storage product at 20 euro per TB (1TB increments), that's replicated on at least two separate RAID clusters at 2 data centers in France. Their English doc page says "all their data centers" but I think it hasn't been updated for their new Amsterdam location. So it may actually include 3 DC's in 2 countries. That's durable too. But, you may need an online.net dedicated server to access it.

    Maybe what you really want is a small west coast US dedi. Those exist though I don't track offers that carefully. @VortexMagnus often has some though.

    Thanked by 2alfred i83
  • alfredalfred Member, Host Rep
    edited January 2017

    Thanks for the suggestions!

    I think I'm leaning more toward the storage VPS side (with backups to GDrive) at the moment, since it seems like a more flexible choice.

    The online.net offerings look very very promising but I'm just worried about throughput from half the world away :)

    CloudBerry looks interesting but I don't want to jump into it because it's a closed source product, and especially not if the stored format is proprietary or non-standard.

  • Online.net c14 is for last ditch archiving where you can survive if it takes a week to get your 500gb back at 10mbps or whatever. You'd want to put more normally accessed stuff somewhere faster. Agreed that distance gets in the way for that.

    I'd never use any closed source product for backup, sorry. You might have to restore on some weird computer when the product is no longer supported, etc.

  • varunchopravarunchopra Member
    edited January 2017

    Why don't you just use GDrive to store backups?

    https://github.com/prasmussen/gdrive

  • alfredalfred Member, Host Rep

    @varunchopra I'd like incremental backups, which isn't really supported on Google Drive? I guess I could back up locally and then sync to GDrive, but I don't have the local disk space at the moment.

  • @alfred That's pretty much what I was hinting at.

    Thanked by 1alfred
  • I use a local luks volume on an old HDD for rsnapshot backup and then just borgbackup to a remote borg 'repo'. Only the changed /added bits are sent over WAN.

    Borgbackup. use it. It covers all the conceivable backup requirements including client-side encryption.

  • Do Borg users like it better than Duplicity?

    Thanked by 1k0nsl
  • Check out https://sia.tech.

    1 TB boils down to $2 pm.

  • varunchopra said:

    Check out https://sia.tech.

    1 TB boils down to $2 pm.

    I looked. That's terrifying. And the $5/TB download cost makes economically unattractive too.

  • @varunchopra said:
    Check out https://sia.tech.

    1 TB boils down to $2 pm.

    Thanks, interesting! Isn't this what Kim.com is trying to invent these days?

  • pechspilz said: Thanks, interesting! Isn't this what Kim.com is trying to invent these days?

    Probably, another thing bound to fail... it's the same concept as Wuala essentially, again.

    I just have a box somewhere with a HW RAID (~8TB usable), run ZFS and deduplication. Stores years of backups for hundreds of servers and has space for years more.

    Fed by rsync pull from the backup server, update only last snapshot and then snapshot again.

    Machine now has 193511 snapshots and Ubuntu still fine :)

    Thanked by 2vimalware alfred
  • @pechspilz said:

    @varunchopra said:
    Check out https://sia.tech.

    1 TB boils down to $2 pm.

    Thanks, interesting! Isn't this what Kim.com is trying to invent these days?

    https://www.reddit.com/r/IAmA/comments/5p5xbp/were_the_team_behind_sia_a_new_type_of_cloud_that/dconv0v/

    Thanked by 1pechspilz
Sign In or Register to comment.