New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Running around mostly at the time, scrambling staff of all areas as AWS does never tell you what the issue is (who knows, could be large scale hack or DC burning, so better engage everyone at first) - time also bad for PST/CA timezone, right over 12:00.
S3 being down did not have massive effect on too many, only when other things started to fail large scale it became more of an issue for even largest users, we are talking here about vast parts of the eco system even internally being unusable.
Many people are pissed at AWS now obviously, executive and their higher up will answer to larger customers within the next day official and these hours informally (already running) and provide some more light. Will be fun.
This did also show some major AWS problems (or rather, shortfalls in API options and who has access to them) in service restore as ELB and auto scaling for the hosted services is utterly useless if you want to add massive traffic within few minutes (try it out, create ELB and add 10 instances, then run load against it immediately, then an hour later when the ELB is scaled).
On the up side our GPU instances survived fine as did EU-WEST-1 failover for most of the things.
Backblaze's block storage is cheap compared to other companies. What do you mean by 'mill LEB storage plan'? Do you mean something like sotrage VPSs?
Burn, Bezos, Burn.
Don't know what Sotrage is. Backblaze B2 is 0.005/GB for storage plus .05/GB for outbound bandwidth, so you get creamed if you actually retrieve stuff. By comparison, OVH object store is 0.011/GB storage and 0.011/GB outbound bw, OVH cloud archive is around 0.002/GB for storage but 0.011/GB bw both inbound and outbound; and LEB storage plans are plentiful in the 0.007/GB-0.010 range with free bw up to some reasonable multiple of the storage capacity. Some like Time4VPS are as low as 0.004. As for raw storage at big providers, Hetzner StorageBox is 0.004 in the 10TB size with lots of free bw, though it's an scp/sftp system rather than an object store. B2 sounds closer to Hubic which is 5 euro/month for 10TB but i/o is capped to 10mbps. B2 is observably even slower than that.
Strange that some very serious sites not avoid single point of failure. Maybe they will learn a bit more after this incident.
Amazon, and CF will still exist- and thrive. So, I doubt it.
@WSS For sure after this incident Amazon will lose some customers. Avg. response time for ticket in Amazon is 3 working days (without extra paid support). Great customer service
That's because they're not willing to fork over the money for overnight, when Prime 2nd day is already paid for.
@WSS Amazon never heard about "keep things simple". Try start using their API. You need to reserve few days to find all important things. Similar as Google documentation.
I've gone through S3, and before Bezos burned me, I was starting to develop apps for the Echo. Now I go out of my way to avoid Amazon. It's served me well, even if it is a little more difficult to make impulse purchases of things I actually need when I need to drive to get it.
I feel like I'm missing a rant or anecdote here, as well as the actual sex act "burned" is euphemizing.
I'll give you the short-short version. Amazon is turning into a flea-market version of eBay. I contacted them several times about illegal counterfeits and software being sold through Amazon Marketplace. The support droids said they'd make a note, but that there was no practice to ever do anything by the time I got the same person three times in so many months.
I got sick of it and sent all of this to good old hefe@amazon, which was then blind forwarded to a subordinate. They gave me a blowoff response for subsequent followups about these items still being for sale, so I sent it again to Jeff, and CC'd a few larger companies anti-piracy contacts.
Within hours, I had lost my top 500 "Trusted Reviewer" status, and all of my reviews were removed as "violating terms". It's not like I was making money for reviewing items- their response to my "Hey, you guys are unwittingly allowing this" being to do nothing, then brush me under the carpet, welp- they don't need my money.
I came here to read some quality shitposting on aws 'true believers'.
I am disappointed, disappointed I tell you.
Well, who believes in AWS these days?
Never thought about outbound traffic. Thanks for pointing that out!
OVH looks very cheap.. gotta check them out. Thanks!
Every startup, every big company out there, I'm working for one and they don't even want to budget even after what happened.
For big companies, I understand. But for startups, isn't DO with the startup plan (something with free DO for few years. forgot the details) or other well-known VPS providers (like linode, vultr, ovh, leaseweb, etc) a better option?
>
All of those you mentioned are completely different to AWS. Do you need raw compute(in which case you may evaluate other providers) or do you need features from the Amazon Ecosystem to build your project?
People need to stop comparing Amazon as compute only, it's not, and there is a WHOLE lot more it offers than just a terminal window with some ram and disk attached.
Complexity kills the cat, believe me, I know.
The reason it claimed to be "online", was that the status icons were hosted on S3 as well, and so their cache was stuck with an outdated ("online") status, because S3 was down...
I wish I was making this up.
Even the big guys make an arse of it. Still waiting for Google to make a monumental mistake though, assuming Google+ isn't one.
"We received a report that G+ is down."
'Wait? Someone's actually using it!?'
Serious question. Why would anyone use S3? They are quite expensive for what you get.
I get why some big enterprises use them. Big enterprises like to work with other big enterprises and are not so price sensitive. For everyone else, I have a hard time finding a good reason.
Amazon doesn't have a monopoly on building reliable systems or some magic reliability fairy dust. This outage only confirms that. More about perceived reliability imo.
AWS is still having the highest uptime of the three.
In 2015 Azure and google had an average of 11 hours downtime, while AWS just a bit over 2.
IWStack was close to the last one, azure, with about 14 hours aggregate.
There is no magic sauce, when Murphy is after you, you'll get it.
These solutions are enormously complex, nobody can have a mitigation plan in place to cope with all possible, scenarios, heck, not even all the imaginable ones.
I'd guess again. I only use SES so I can comment only there but it started working for me ~5 hours after when they said S3 was fixed.
I also liked how I couldn't just migrate to their other zone without creating a new account there.
Well, it depends how you calculate downtime. It was one service in one location.
True, one of the worst possible failures, but still, most other things on 4 continents were up.
What I still don't understand is, why would they build their status page on top of S3? They should at least build their status page for S3 on someplace else. Plus, they should have replicated thoes data to other regions to minimize the possibility of status page not working because of a system error.
Because it was fucking hilarious- that's why!
Why are there so many people focusing on their status page rather than the actual service. The actual outage is far more serious that what some status page says.
Do you need a status page to tell you if your service is not working?
It is the only consolation when it doesn't...