Digital Ocean - Monitoring

jar

@Kris said:
Maybe a bit too taxing to the system?

Digital Ocean support and management panels down 40 minutes now

{"id":"service_unavailable","message":"Failed to forward the request you made, please try again."}

Ah, someone's noticed : https://status.digitalocean.com/

Oh man that's a bit long. I just woke up but dang, this is a postmortem I'm bringing popcorn to

No idea what happened but to be out that long, no way anyone is having a good time right now.

dedicados

willie said: Linode, multiple times.

good to know, thanks

PrestigeWS

Wanted to try out the monitoring thing and sadly its still down ..!

jar

@PrestigeWS said:
Wanted to try out the monitoring thing and sadly its still down ..!

The good(ish) news is everyone is on top of it and that it is expected to be back up fairly soon.

Lee

@jarland said:
The good(ish) news is everyone is on top of it and that it is expected to be back up fairly soon.

I am loosing thousands of $$$$'s every hour because of this....

Hxxx

DO under maintenance... Thats a long ass maintenance.

HBAndrei

@TimeWillTell said:
Does that mean they will start charging for over usages of bandwidth which they never did because no monitor of traffic

Doesn't look like this agent would be the way to go about it:

@jarland said:

@Luke007 said:
With monitoring in place, is this the end of unmetered bandwidth in Digital Ocean?

I wouldn't take this as an official statement, but I am unaware of any effort to push something like that forward.

I would also consider it flawed since do-agent is open source, so if we relied on it's numbers for billing I'd call it a laughable effort. One fork that reports false numbers and game over

Hxxx

@jarland Give my thanks to the DO team. Excellent tool.

jar

@Hxxx said:
@jarland Give my thanks to the DO team. Excellent tool.

Kind words always appreciated on a rough day

WSS

@Hxxx said:
@jarland Give my thanks to the DO team. Excellent tool.

He is quite a nice person, don't denegrate, brah!

reetwood

is jarland the big boss man of digital ocean? well your last meetup pizza's sucked, what i'd like is better pizza

jar

@reetwood said:
is jarland the big boss man of digital ocean? well your last meetup pizza's sucked, what i'd like is better pizza

Just a lowly team lead here, but I can't work at a company and not consider it's product to be my baby

Also +1 for better pizza.

reetwood

damn, i was hoping you had some weight. maybe enough weight to change them pizzzas. i guess not, well i guess i can't really complain, it was free food. but this time, im going to drink as many beers as i can.

also, would it be ok, if i just snuck a box onto the DO office network? like i left an rpi connected, but maybe something more beefy in a cupboard? cos i figured you guys are lostcost providers. so it would be right up in your neighbourhood. Do you guys use RCN? or what?

jar

reetwood said: would it be ok, if i just snuck a box onto the DO office network?

Just don't set all of the iPads to amazon.com. Apparently I'm the only one that thought that was funny.

WSS

@jarland said:

]> Just don't set all of the iPads to amazon.com. Apparently I'm the only one that thought that was funny.

I load a semi-stock Google [experience] onto all $50 Kindles before I gift them on. No Amazon shit included- just the kernel and bootloader. I've had no complaints to date.

reetwood

you mean kindle fire? your a real tight ass

WSS

willie

Lee said: I am loosing thousands of $$$$'s every hour because of this....

One MILLLLLLLLLION dollars!!

Kris

https://www.digitalocean.com/company/blog/update-on-the-april-5th-2017-outage/

tl;dr:

At 10:24 AM EDT on April 5th, 2017, we began to receive alerts that our public services were not functioning. Within three minutes of the initial alerts, we discovered that our primary database had been deleted. Four minutes later we commenced the recovery process, using one of our time-delayed database replicas. Over the next four hours, we copied and restored the data to our primary and secondary replicas. The duration of the outage was due to the time it took to copy the data between the replicas and restore it into an active server.

At 3:20 PM EDT the primary database was completely restored, and no data was lost.

Cause :

The root cause of this incident was a engineer-driven configuration error. A process performing automated testing was misconfigured using production credentials. As such, we will be drastically reducing access to the primary system for certain actions to ensure this does not happen again.

Ouch. I cannot imagine the feeling of that dev - config error wiped out primary production database.

It looked bad when I was getting a 406 on status.digitalocean.com this morning including everything else, but boy that was rough. Great turn around on the RFO.

Maybe DO should CNAME status. to a StatusPage.io instead of keeping it within network.

reetwood

well hot diggity, lucky i rely on AWS for my critical shit.

jar

Kris said: Maybe DO should CNAME status. to a StatusPage.io instead of keeping it within network.

It's actually out of network but it definitely needs some love regardless

WSS

The fact that ErrorPageAsaService makes me want to stab people in the head. That's not even the millenials, yet.

jar

@reetwood said:
well hot diggity, lucky i rely on AWS for my critical shit.

Yeah good thing they never have outages!

Kris

WSS said: The fact that ErrorPageAsaService makes me want to stab people in the head. That's not even the millenials, yet.

Sometimes though, warranted.

Not sure how it took out the status page, but others on YC saw it as well.

bap

@Lee said:

@jarland said:
The good(ish) news is everyone is on top of it and that it is expected to be back up fairly soon.

I am loosing thousands of $$$$'s every hour because of this....

just call a lawyer..