New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Do your own backups (AWS deleted person's 10-year account and all data without warning)
I've gone with a title that's a slightly less popular take on this story, but this story does nonetheless show how important keeping your own backups and practising recovery is...
https://seuros.com/blog/aws-deleted-my-10-year-account-without-warning/
Comments
Do your own OFFSITE backups
UPD. Actually should be more like multi vendor
https://arstechnica.com/gadgets/2024/05/google-cloud-explains-how-it-accidentally-deleted-a-customer-account/
We always have offsite backups for any of our important data!
Very well written and the site design is awesome.
No, you basically click-bated us.
At first I thought "unholy fuck, I hope ralf somehow gets his data back!". But now, having glanced over "the message" you wanted to propagate, i.e. the linked article, all I say is this:
One would hope that that idiot finally has learned his lesson. One does NOT rely on only 1 provider!, period. No matter how big that provider is.
One always has (at least) two backups in two locations, preferably at least a couple of hundred miles apart and with different providers. Plus ideally a disaster plan (e.g. what if one is struck by disaster?).
AWS is for burning money and losing your backups!
I've worked with a number of clients over the years that have had their faith totally within the cloud ecosystem they are using. It's infuriating at first, but quite easy to snap them out of if you run a proof-of-concept with a blank-ish system.
I'm quite old fashioned though... I still like having semi-regular backups on magnetic tape. Unless there is an EMP, recovery from any catastrophe is a loooot easier.
@ralf , I've edited the thread title in order to make it less misleading
For the historical record, the original title was:
That font is giving me eye cancer but thanks for sharing.
Oh noes! I think I've confused everyone here...
Just to be clear, this isn't my article. I have no connection with the author, I just found the article on HackerNews and came to a completely opposite conclusion but still found it interesting, which is why I thought I'd share it.
No, no, no. It's not my data.
Exactly, the article's point was "AWS deleted all my data, how can a company be so crap?" and I wanted to share it here (and apparently had a typo in the title, maybe confusing the issue) as "Do your own backups"
If we're changing the bit in brackets, we should just remove it entirely.
The bit in brackets was the title of the article as posted by the author of the article, I wanted to share it as "Do your own backups", because that should have been the key takeaway IMHO.
database backup everyday, files backup every week. all sent to Google
I've edited the title again:
Then a title like "Do your own backups properly - or else [[title of linked article]!)" would have been better and way less misleading (and as I perceived it click-baiting).
That's why I thanked @angstrom. He made the point somewhat more clear.
Yeah, no worries. It's all good. I was tired and just about to go to bed, but thought I'd share it just before I did. My explanatory paragraph was a jumble of incoherence as well! If I was more awake, it'd have been better...
still, appreciate the post very much
Cloud backup is fine. Just have more than one vendor.
I backup all of our important stuff at work to Backblaze B2 as full VM snapshots, and then app backups of important files and databases to Backblaze B2 as well as to Cloudflare R2.
I don't necessarily doubt the author's version of events (even if he becomes very emotional towards the end), and it seems that the automated nature of AWS left very little room for timely and useful human intervention, but the initial triggering incident nevertheless seemed to be a failure of the author to respond to the verification request within the requested time frame (or so it seems to me)
This isn't at all to excuse AWS -- not at all -- but the story then becomes involved and complex
In any case, yes, the lesson is to keep one's backups at a provider that is independent of the provider that is hosting the original files
He did put all eggs in the same basket, contrary to what he wrote. Backups should always be external.
I’d replace ’at a’ with ’at several’
It's kind of a non-story right? If your data matters make sure you take care of it. He's mad they deleted his shit (sure, I would be) and so he angryblogged about it.
If the bit about an engineer accidentally his data, well that sucks but it doesn't really matter since the data's gone regardless and really who hasn't accidentally somebody's data at some point?
Generally if you didn't take backups seriously yet it's because you haven't experienced needing backups and not having them yet. Once that happens you start taking it seriously.
If you're thick you might have to repeat the learning process for things like testing restores.
I did lol at how he has customers totallying $400k/mo in aws billing and they're all ready to jump ship with him, but whatever.
EXACTLY
Off-side isn't enough. Several vendors is needed. Not sure how I missed this.
Pretty terrifying they just randomly demand KYC on a ten year customer. I guess there was a missed payment problem, but they could have at least given him some options to get the bill paid up before demanding invasive PII. Guessing they just wanted that in order to send him to collections and not actually help him.
Dude, you didn't read the story right? He did almost everything right, multi-region backups and everything. AWS fucked up. The followup story is more interesting: https://www.seuros.com/blog/aws-restored-account-plot-twist/
Account restored, when the shit hit the fan to the CEO. But "My data is mostly back - I lost the last weeks of work because the spaceship wasn’t installed and encryption keys weren’t synced. But losing weeks is better than losing everything."
Yes of course AWS fucked up, but nothing about the story surprised me and hopefully he's expanded outside of AWS so next time it won't be so devastating. Even the mechanics behind the issue isn't really surprising, but maybe I'm just extra cynical.
Anyway it's good he's back online and a shame he had go thru a month of hell. Score again for social media saving the day. Isn't that scary too?
edit: thanks for the followup, I hadn't seen it.
i agree. backing up with same company different datacenter is not called a external backup as account operated is same. If someone really does want to keep is with same company different dc, then create new account in with your family member's name/card and use it. Though is is applicable for huge providers like aws/google etc. For small LE provides better keep backup with totally different provider. We never know the entire DC can vanish!
valid
to an extent
"Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t." he did.
I've tried both firefox and chromium, but I don't see any story text on that page. Just a headline and a large graphic.
might be font issue. They're using some weird font (Serif)
This kind of operation is very frightening.