Setting up advanced HTTP Flood protection

PrincessOfCats

After multiple massive HTTP floods on a friend's site I was hosting (I already had protection for other attacks), I have been working on a cookie/js verification based on what CF has. As much as I would like to use CF, its let me down a few too many times in the past.

So far, I understand what I am doing with the cooke/js verification. The main issue is that the page needs to be displayed before anything on the site is displayed.

So far, I have only had one idea, and that is requiring a valid cookie to access any page other than the default page where I do the cookie/js verification. Is there a better way to do this?

PrincessOfCats

edit: nvm, figured it out.

There is actually a nice nginx module to do this: https://github.com/kyprizel/testcookie-nginx-module

I currently use a haproxy -> varnish setup, but I can slip nginx in between to do the verification.

What Ive decided:

Haproxy currently handles the rate limiting swimmingly. When it gets to a certain number of requests/time, it swaps the ip to point at a different backend (in this case, nginx). Nginx will do the testcookie to see if stuff is allright, if it is, it will allow the request. If not, it will display an error page.

tomsfarm

You could take a look at nginx testcookie it might work for you

 http://kyprizel.github.io/testcookie-nginx-module/

aglodek

@StartledPhoenix said: …I currently use a haproxy -> varnish setup, but I can slip nginx in between to do the verification… Haproxy currently handles the rate limiting swimmingly.

Interesting setup, but have a few questions if you don't mind:

• why did you choose HAProxy and not Varnish at the front?
• wouldn't it be more efficient to do this part using iptables?

When it gets to a certain number of requests/time, it swaps the ip to point at a different backend (in this case, nginx). Nginx will do the testcookie to see if stuff is allright…

• and by allright, I take it you mean traffic from users identified with an appropriate cookie? What about first time visitors?

…if it is, it will allow the request. If not, it will display an error page.

• wouldn't it be better to route the flood to a nullrouted IP or some other "blackhole" solution?

PrincessOfCats

@aglodek said:

Varnish has no https. I use HaProxy as a SSL terminator as well as a rate limiter. Nginx's rate limiter is not as good as I hoped and it fell flat on its face a few times (the requests just started timing out even though max connections were above what was being done)

Iptables causes ips using NAT (Universities/etc especially, who decide to use a single IP for all outbound student communcations. Some use multiple for wifi/ethernet or whatever) to trip and fall flat on their face without any reason. I used to do rate limiting only with haproxy. This was one of the reasons that I wanted to do cookie/js verification. People who share IPs can still access the site without getting rate limited.

With this method, I can display a page for who to contact to tune some stuff in case they get banned accidentally. I highly dislike the fact that people have no idea where to go to get unbanned because they are banned.

For the cookies part, read the description, the page sets the cookie, and if there is no valid cookie, nginx goes back to the cookie-setting page for a limited number of times.

Side note: Ideas about setup were borrowed from Reddit rate limiting and CF.

aglodek

@StartledPhoenix said:

Simple yet ingenious Thanks!

PrincessOfCats

@aglodek said:

Also, if some people are DDOSing you from a school (This was actually the cause of me needing to rate limit...) and the IT Department doesn't give a crap, you won't end up blocking everyone in the school, just those that don't pass the JS/Cookie test.

This method can be bypassed by specialized 'bots' that some booters have these days (they actually do the JS/Cookie stuff!), but I plan on further working on it to get some more protection when time comes.

Chances are, by the time those become extremely popular, my DDOS protection will have to be upped before I start tinkering with the HTTP flood protection

If you want to test on Ubuntu (sorry centos users....), see https://launchpad.net/~phoenix-o/+archive/ubuntu/nginx-security/+packages

It will require apt pinning due to debian revisions.

Nomad

And maybe you can use fail2ban to monitor nginx logs. For http flood it might work.