New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Amazon S3 for Backups
Hello,
I am wondering how many of you here are using Amazon S3 for their daily "Incremental backups" for the server. If so, can you give me some idea's on what tool / method you are using?? And how effective that has been for you??
I looked around and found few tutorials but that seems to be dated back to 2012. And a few recent ones only talk about simple backups. (Not incremental ones)
Thanks
Comments
I use duplicity with the S3 backend.
Duplicity has been very effective for me in full system backups. I personally like how I don't have to trust Amazon as all my backups are signed and encrypted.
"I like how I should not trust Amazon.."
Well I do trust Amazon - but trust isn't required for my relationship. :P
Encrypt, don't trust. At least for now using encryption is legal in most parts of the world. Though governments want to make it illegal.
Want to help on how it's done with duplicity? Any tutorial out there ??
Maounique, did you accidentally post using the wrong account? :-)
Nah, if it was Mao the post would be a lot longer
I use this free tool: s3tools.org
Example of synching a folder on a server
/home/site1.com/excludes.files
For my daily incremental backups, I recently switched from rsnapshot (using rsync) and S3 to attic (deduplicating backup) and Google Storage Nearline. Not only did the size of my total backup shrink by multiple magnitudes, but also is Nearline cheaper than S3.
@zeitgeist - what are you paying per GB storage and per GB bandwidth?
@MarkTuner -
Source: TechRepublic
Glacier and Nearline are both attractive to store data in...for example, 2TB is $20/month and it's high quality storage, not some LEB.
The problem is that if you ever want to get the data out...it's extremely expensive. 2TB = $180 of outgoing bandwidth. Ugh.
@raindog - Our S3 compatible storage has packages available which heavily drops the cost:
1TB Storage
1TB Bandwidth
$19.95/month
2TB Storage
2TB Bandwidth
$34.95/month
I've heard rumors of this service but no one will send me a link to it :-)
It's also not quite the same. Amazon and Glacier are huge companies with PhDs designing their infrastructure. Delimiter might be an ace provider (I've never used them) but they're not quite in the same league in terms of safety/reliability/will be there in five years/etc.
@raindog308 - Without question there is a size issue, but Delimiter/Yomura have been around nearly a decade. S3 storage isn't rocket science, really the only consideration is survival. Will Yomura survive the next decade, more than likely. Its got a good few people bankrolling it and is more profitable than Amazon.
PM me your email address and I'll get you setup so you can test it.
That's quite a bold statement to make. I also highly doubt it's true.
Amazon is notorious for not turning a profit, just continuing to re-invest. But AWS specifically is quite profitable, so @MarkTurner is a tad off there.. http://www.businessinsider.com/aws-earns-1-billion-a-year-on-6-billion-in-profit-2015-4
I was playing the semantics game, I specifically said Amazon rather than AWS. Amazon hasn't ever turned a profit, I didn't say turnover
The advantage of Nearline over Glacier is that you can access Nearline just like "normal" storage... there is no waiting time involved. I am also a big fan of the gsutil tool... gsutil rsync is very easy to use.
Is this service not offered publicly yet? If no, then I'm not really interested in beta-testing.
If it is production-ready, then can you post a link?
This is my script that is ran nightly, it's normally deployed to my servers using SaltStack so you'll see
grains
- just replace them with your own values.I can't say where I learned about duplicity, most I've picked from the manual pages and Google search results.
https://gist.github.com/Silvenga/76dc9ee36ad476f4b37f
I just use Tarsnap
I love Tarsnap but it's very expensive (not surprising since it's layered on top of AWS).
Thank you so much
Just needed a starting point. This will do.
I have been also looking at Salt for doing a lil automation to my setup. Not there yet, but it's in the list
Delimiter finally announced it this morning, so here's the link:
http://www.delimiter.com/landingpage/objspace/
Devil's advocate, since many use Amazon Glacier for self-healing, multiple-datacenter replication and mandatory object signing, government level encryption, what's Objspace's guarantee of object durability and security? I don't really see a comparison of apples to apples with Objspace and "archival services such as Glacier and Nearline".
@Silvenga - clearly if you are looking for ultimate durability then anyone running this type of service on SSD/HDD is going to be a bad choice. Glacier's secret sauce has been said to be tape or optical, so onboard to HDD then offload to tape/optical for later retrieval. They haven't given a clear statement on this. But without question that is a more durable product at a substantially higher price.
However during the testing of the product, it was apparent that many customers were using Glacier to reduce costs rather than because they wanted their backups to last 100 years or wanted multisite redundancy (though that will be coming for Delimiter). Customers wanted a way to backup their local files, so instant access wasn't critical but pricing was.
Other users wanted a object store for images, mailboxes and most commonly VM templates/snapshots. For that market, its well positioned.
500GB for free! Can't beat that price.
(The price does set itself correctly if you move the slider to 1TB and back to 500GB, but when you first load the page it shows $0).
@raindog308 - What browser/OS is that on?
@MarkTurner same thing happens to me. I'm on Chrome/Mac