Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Secure Dragon down
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Secure Dragon down

NickkNickk Member
edited December 2012 in General

My monitoring says both VPSs have been down for the last two and a half hours.

I'm surprised there is no mention of this already. Are other people up?

Comments

  • Mine is working, no problem.

  • sleddogsleddog Member
    edited December 2012

    Came back a few minutes ago... looks like networking issue.

  • I was down for a while, must have been something network related because I just logged in and my uptime is 35 days.

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    We're back online and I am beyond pissed at this point. Here's what happened:

    2AM-4AM: Mitigating DDOS attacks for a client that decided NOT to use the DDOS Protected IP he purchased fully aware as soon as his website went online it would result in large DDOS attacks.
    4:15AM: I went to sleep thinking the nullroute would keep the attacks at bay.
    4:37AM: DDOS attacks getting larger and hitting random targets on our network until both of our routers go offline (both interfaces on both routers lost their VRRP IPs for some reason),
    7:24AM: I wake up to a sea of alerts on my phone. A quick config rebuild and reboot bring both routers and our network back online.
    7:45AM: I'm directed to a thread of the same client bitching about our DDOS protection and the downtime he caused for not using our DDOS filtered IPs.

  • Sounds like there's not much more you could have done, @KuJoe. Keep up the good work!

  • Shame these things happen.

    I'd get all the routers @KuJoe DDoS protected and look at how the protection is rolled out. Core stuff has to stay up, but you know that. Yeah, I know. probably can't do that easily.

    But, now the world knows what it takes to make your operation go offline. :(

  • I hope you hit the big red terminate button on this guys account.

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    The client has been dealt with so I don't expect any more attacks relating to their website. They popped off a not-so-polite ticket about the downtime they caused so I sent them elsewhere. Funny thing is the size of the attacks could have easily been mitigated by our protection but they chose not to use it (which is their choice, but then choosing to complain about it publicly and then open a ticket complaining about it was their last mistake).

    It's unfortunate that the protected IPs are tunneled in (and pricey) or I'd have all of our IPs protected.

  • Want to trade DDoSed clients for habitual DMCA offenders? ;-)

  • We're back online and I am beyond pissed at this point. Here's what happened:

    You have every reason to be pissed. We noticed our VPS's coming an going, so it loocked network related. I am about 16 ahead of you time-zone wise, so it was my peak time so to speak.

    But you did the best you could do..!

  • KuJoeKuJoe Member, Host Rep

    I'll take DMCA takedowns over DDOS attacks any day. I'm glad it's only the 13th because our 95th percentile isn't looking to good after last night's attacks. :(

  • serverbearserverbear Member
    edited December 2012

    We caught about 3 hours of downtime:

    http://serverbear.com/9783/serverdragon#tampa-fl-usa

  • KuJoeKuJoe Member, Host Rep

    Upon further investigating it looks like our backup switch isn't doing so good either. I'll be going to the DC tomorrow to see why we have downed ports on it. :(

  • KuJoeKuJoe Member, Host Rep

    @serverbear Yup, that looks similar to what Pingdom and NodePing are reporting. Our 99.9% SLA is out the window for the month now. :(

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    Priceless... I removed the nullroute for the target IP and the DDOS attack from 2AM is still on-going. Today is going to be a fun day indeed. ~3 hours of sleep and an important meeting today with my boss and my department head. :(

  • Pity you @KuJoe.

    Null the IP and let it fester for 24 hours. The typical wait and see dance routine.

    Overdue for escalation to knock these sorts of attacks out. Unacceptable all these years that such a simple exploit of sorts remains as effective as ever.

  • jarjar Patron Provider, Top Host, Veteran

    Sounds like a fun one indeed. Holler if there's anything we can do for you brother.

  • Just reach home and found out the down time. My backup server was down for nearly 3 hours...

  • @seikan said: My backup server was down for nearly 3 hours...

    Good thing it is a backup!

  • How's your backup's backup doing? :)

    N + 1 = 3 for sanity sake

  • SecureServers filtered IP is working pretty good against syn's.

    @kujoe, how do you nullroute IPs at Vyatta? Got an eBGP session with the datacenter?

  • KuJoeKuJoe Member, Host Rep

    @apollo15 said: @kujoe, how do you nullroute IPs at Vyatta? Got an eBGP session with the datacenter?

    Yup, just doing BGP tagging.

Sign In or Register to comment.