New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Mine is working, no problem.
@KuJoe
Came back a few minutes ago... looks like networking issue.
I was down for a while, must have been something network related because I just logged in and my uptime is 35 days.
We're back online and I am beyond pissed at this point. Here's what happened:
2AM-4AM: Mitigating DDOS attacks for a client that decided NOT to use the DDOS Protected IP he purchased fully aware as soon as his website went online it would result in large DDOS attacks.
4:15AM: I went to sleep thinking the nullroute would keep the attacks at bay.
4:37AM: DDOS attacks getting larger and hitting random targets on our network until both of our routers go offline (both interfaces on both routers lost their VRRP IPs for some reason),
7:24AM: I wake up to a sea of alerts on my phone. A quick config rebuild and reboot bring both routers and our network back online.
7:45AM: I'm directed to a thread of the same client bitching about our DDOS protection and the downtime he caused for not using our DDOS filtered IPs.
https://my.securedragon.net/announcements.php?id=239
Sounds like there's not much more you could have done, @KuJoe. Keep up the good work!
Shame these things happen.
I'd get all the routers @KuJoe DDoS protected and look at how the protection is rolled out. Core stuff has to stay up, but you know that. Yeah, I know. probably can't do that easily.
But, now the world knows what it takes to make your operation go offline.
I hope you hit the big red terminate button on this guys account.
The client has been dealt with so I don't expect any more attacks relating to their website. They popped off a not-so-polite ticket about the downtime they caused so I sent them elsewhere. Funny thing is the size of the attacks could have easily been mitigated by our protection but they chose not to use it (which is their choice, but then choosing to complain about it publicly and then open a ticket complaining about it was their last mistake).
It's unfortunate that the protected IPs are tunneled in (and pricey) or I'd have all of our IPs protected.
Want to trade DDoSed clients for habitual DMCA offenders? ;-)
You have every reason to be pissed. We noticed our VPS's coming an going, so it loocked network related. I am about 16 ahead of you time-zone wise, so it was my peak time so to speak.
But you did the best you could do..!
I'll take DMCA takedowns over DDOS attacks any day. I'm glad it's only the 13th because our 95th percentile isn't looking to good after last night's attacks.
We caught about 3 hours of downtime:
http://serverbear.com/9783/serverdragon#tampa-fl-usa
Upon further investigating it looks like our backup switch isn't doing so good either. I'll be going to the DC tomorrow to see why we have downed ports on it.
@serverbear Yup, that looks similar to what Pingdom and NodePing are reporting. Our 99.9% SLA is out the window for the month now.
Priceless... I removed the nullroute for the target IP and the DDOS attack from 2AM is still on-going. Today is going to be a fun day indeed. ~3 hours of sleep and an important meeting today with my boss and my department head.
Pity you @KuJoe.
Null the IP and let it fester for 24 hours. The typical wait and see dance routine.
Overdue for escalation to knock these sorts of attacks out. Unacceptable all these years that such a simple exploit of sorts remains as effective as ever.
Sounds like a fun one indeed. Holler if there's anything we can do for you brother.
Just reach home and found out the down time. My backup server was down for nearly 3 hours...
Good thing it is a backup!
How's your backup's backup doing?
N + 1 = 3 for sanity sake
SecureServers filtered IP is working pretty good against syn's.
@kujoe, how do you nullroute IPs at Vyatta? Got an eBGP session with the datacenter?
Yup, just doing BGP tagging.