Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Amazon S3 for Backups
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Amazon S3 for Backups

UmairUmair Member

Hello,

I am wondering how many of you here are using Amazon S3 for their daily "Incremental backups" for the server. If so, can you give me some idea's on what tool / method you are using?? And how effective that has been for you??

I looked around and found few tutorials but that seems to be dated back to 2012. And a few recent ones only talk about simple backups. (Not incremental ones)

Thanks

«1

Comments

  • SilvengaSilvenga Member
    edited June 2015

    I use duplicity with the S3 backend.

    duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup.

    Duplicity has been very effective for me in full system backups. I personally like how I don't have to trust Amazon as all my backups are signed and encrypted.

    Thanked by 1Umair
  • @Silvenga said:
    I use duplicity with the S3 backend.

    Duplicity has been very effective for me in full system backups. I personally like how I don't have to trust Amazon as all my backups are signed and encrypted.

    "I like how I should not trust Amazon.."

  • Issam2204 said: "I like how I should not trust Amazon.."

    Well I do trust Amazon - but trust isn't required for my relationship. :P

  • rds100rds100 Member
    edited June 2015

    Encrypt, don't trust. At least for now using encryption is legal in most parts of the world. Though governments want to make it illegal.

  • UmairUmair Member

    @Silvenga said:
    I use duplicity with the S3 backend.

    Want to help on how it's done with duplicity? Any tutorial out there ??

  • raindog308raindog308 Administrator, Veteran

    rds100 said: Encrypt, don't trust. At least for now using encryption is legal in most parts of the world. Though governments want to make it illegal.

    Maounique, did you accidentally post using the wrong account? :-)

  • rds100rds100 Member
    edited June 2015

    raindog308 said: Maounique, did you accidentally post using the wrong account? :-)

    Nah, if it was Mao the post would be a lot longer :)

  • I use this free tool: s3tools.org

    Example of synching a folder on a server

    s3cmd sync  --delete-removed --exclude-from /home/site1.com/excludes.files /home/site1.com/public_html/ s3://bucket-name/backup/site1/public_html/
    

    /home/site1.com/excludes.files

    folder1/cache/*
    folder2/log/*
    folder3/templates_c/*
    folder4/tmp/*
    folder5/upgrade/*
    
  • For my daily incremental backups, I recently switched from rsnapshot (using rsync) and S3 to attic (deduplicating backup) and Google Storage Nearline. Not only did the size of my total backup shrink by multiple magnitudes, but also is Nearline cheaper than S3.

  • @zeitgeist - what are you paying per GB storage and per GB bandwidth?

  • TACServersTACServers Member
    edited June 2015

    @MarkTuner -

    This month, Google has opened the beta of Nearline Storage in the Google Cloud Platform. Nearline is a new storage class that has slightly lower availability and slightly higher latency than standard storage, but at a much cheaper rate -- Nearline Storage is $0.01 GB/month, whereas standard storage is $0.026 GB/month. Nearline operates using the same APIs and models as the rest of Cloud Platform storage, and converting existing buckets to Nearline is a trivial operation.
    
    Nearline is positioned as a pseudo-competitor to Amazon's Glacier service, but with crucial differences. Glacier data storage has a retrieval time measured in hours, as opposed to about three seconds with Nearline. Glacier also has steep penalties -- $0.03 per GB -- for data deleted before 90 days has passed, whereas Nearline bills usage for a full 30 days, even if data is deleted before that time.
    
    Amazon allows Glacier storage users to retrieve up to "5% of your average monthly storage, pro-rated daily, for free each month," but the rates for users who want their data quickly, and above the free tier, should prepare to pay, according to this labyrinthine and confusing formula. Google's billing is much more streamlined, with egress rates worldwide (excluding China and Australia) at $0.12 per GB up to the first TB, with lower rates after.
    

    Source: TechRepublic

    Thanked by 1MarkTurner
  • raindog308raindog308 Administrator, Veteran

    Glacier and Nearline are both attractive to store data in...for example, 2TB is $20/month and it's high quality storage, not some LEB.

    The problem is that if you ever want to get the data out...it's extremely expensive. 2TB = $180 of outgoing bandwidth. Ugh.

    Thanked by 1netomx
  • @raindog - Our S3 compatible storage has packages available which heavily drops the cost:

    1TB Storage

    1TB Bandwidth

    $19.95/month

    2TB Storage

    2TB Bandwidth

    $34.95/month

    Thanked by 1dragon2611
  • raindog308raindog308 Administrator, Veteran
    edited June 2015

    MarkTurner said: Our S3 compatible storage has packages available which heavily drops the cost

    I've heard rumors of this service but no one will send me a link to it :-)

    It's also not quite the same. Amazon and Glacier are huge companies with PhDs designing their infrastructure. Delimiter might be an ace provider (I've never used them) but they're not quite in the same league in terms of safety/reliability/will be there in five years/etc.

  • @raindog308 - Without question there is a size issue, but Delimiter/Yomura have been around nearly a decade. S3 storage isn't rocket science, really the only consideration is survival. Will Yomura survive the next decade, more than likely. Its got a good few people bankrolling it and is more profitable than Amazon.

    PM me your email address and I'll get you setup so you can test it.

  • MarkTurner said: Its got a good few people bankrolling it and is more profitable than Amazon.

    That's quite a bold statement to make. I also highly doubt it's true.

  • @mpkossen said:
    That's quite a bold statement to make. I also highly doubt it's true.

    Amazon is notorious for not turning a profit, just continuing to re-invest. But AWS specifically is quite profitable, so @MarkTurner is a tad off there.. http://www.businessinsider.com/aws-earns-1-billion-a-year-on-6-billion-in-profit-2015-4

    Thanked by 1mpkossen
  • mpkossen said: That's quite a bold statement to make. I also highly doubt it's true.

    mikeyur said: Amazon is notorious for not turning a profit, just continuing to re-invest. But AWS specifically is quite profitable,

    I was playing the semantics game, I specifically said Amazon rather than AWS. Amazon hasn't ever turned a profit, I didn't say turnover :)

  • @raindog308 said:
    Glacier and Nearline are both attractive to store data in...for example, 2TB is $20/month and it's high quality storage, not some LEB.

    The problem is that if you ever want to get the data out...it's extremely expensive. 2TB = $180 of outgoing bandwidth. Ugh.

    The advantage of Nearline over Glacier is that you can access Nearline just like "normal" storage... there is no waiting time involved. I am also a big fan of the gsutil tool... gsutil rsync is very easy to use. :)

  • raindog308raindog308 Administrator, Veteran

    MarkTurner said: PM me your email address and I'll get you setup so you can test it.

    Is this service not offered publicly yet? If no, then I'm not really interested in beta-testing.

    If it is production-ready, then can you post a link?

  • Umair said: Want to help on how it's done with duplicity? Any tutorial out there ??

    This is my script that is ran nightly, it's normally deployed to my servers using SaltStack so you'll see grains - just replace them with your own values.

    I can't say where I learned about duplicity, most I've picked from the manual pages and Google search results.

    https://gist.github.com/Silvenga/76dc9ee36ad476f4b37f

    Thanked by 1Umair
  • sinsin Member

    I just use Tarsnap

  • raindog308raindog308 Administrator, Veteran

    sin said: I just use Tarsnap

    I love Tarsnap but it's very expensive (not surprising since it's layered on top of AWS).

  • UmairUmair Member

    Thank you so much :)
    Just needed a starting point. This will do.
    I have been also looking at Salt for doing a lil automation to my setup. Not there yet, but it's in the list :)

  • raindog308 said: Is this service not offered publicly yet? If no, then I'm not really interested in beta-testing.

    Delimiter finally announced it this morning, so here's the link:
    http://www.delimiter.com/landingpage/objspace/

  • MarkTurner said: Delimiter finally announced it this morning

    Devil's advocate, since many use Amazon Glacier for self-healing, multiple-datacenter replication and mandatory object signing, government level encryption, what's Objspace's guarantee of object durability and security? I don't really see a comparison of apples to apples with Objspace and "archival services such as Glacier and Nearline".

  • @Silvenga - clearly if you are looking for ultimate durability then anyone running this type of service on SSD/HDD is going to be a bad choice. Glacier's secret sauce has been said to be tape or optical, so onboard to HDD then offload to tape/optical for later retrieval. They haven't given a clear statement on this. But without question that is a more durable product at a substantially higher price.

    However during the testing of the product, it was apparent that many customers were using Glacier to reduce costs rather than because they wanted their backups to last 100 years or wanted multisite redundancy (though that will be coming for Delimiter). Customers wanted a way to backup their local files, so instant access wasn't critical but pricing was.

    Other users wanted a object store for images, mailboxes and most commonly VM templates/snapshots. For that market, its well positioned.

    Thanked by 1Silvenga
  • raindog308raindog308 Administrator, Veteran

    MarkTurner said: Delimiter finally announced it this morning, so here's the link: http://www.delimiter.com/landingpage/objspace/

    500GB for free! Can't beat that price.

    image

    (The price does set itself correctly if you move the slider to 1TB and back to 500GB, but when you first load the page it shows $0).

    Thanked by 1MarkTurner
  • @raindog308 - What browser/OS is that on?

  • 17brownj17brownj Member, Host Rep

    @MarkTurner same thing happens to me. I'm on Chrome/Mac

    Thanked by 1MarkTurner
Sign In or Register to comment.