Someone with NGINX Knowledge/IPTables come here.

Nexus

Hey all, I am going to just keep this simple and sweet.

I am looking for a script or maybe a nginx config similiar to this:

imit_req_zone $binary_remote_addr zone=pw:5m rate=1r/s;
limit_req zone=pw burst=5 nodelay;

The problem is, I want nginx, or iptables or whatever you can help me with, to ban that refreshing ip for XX amount of minutes.

Is this possible with nginx, or iptables? I know I cannot fully stop blown out real DDOS's, but I can try. I am using http://deflate.medialayer.com/ as well. But this script runs a cron every minute, I would have to change this to run every 10seconds. (Not sure if that would be a good idea vs performance wise)

I am asking this because, some people think they're "Pro" when they run a stupid HTTP flood tool on my site and it just floods it with $_GET HTTP Requests, (thousands/millions) and just times out PHP and makes the site seem "down" for 20seconds or whatever; until they stop. I am looking for a way to stop them once they hit around 20 consecutive requests. (Whether it's $_POST or $_GET or whatever) anything.

Thanks have a wonderful day.

cosmicgate

Rm -rf *

Maounique

If they request same pages, you can run some cache. That will at least spare the IO hammering if you have enough memory.
M

seikan

Maybe you can use rate limit in iptables to block this. The following rules block an IP with 20 hits within 10 seconds to port 80.

iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --set --name DDOS
iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 10 --hitcount 20 --rttl --name DDOS -j DROP

klikli

@cosmicgate said: Rm -rf *

-bash: Rm: command not found

PS: Relax, the bash is in my mind.

Maounique

indeed, capitalization matters in Linux
M

winston

@cosmicgate I don't even see how that was funny in this case..

Nexus

@seikan My server is down atm, so I cannot test those iptables. But will test them asap!

But would that help against HTTP Flood requests? Specifically this freeware program released by a university to study and help http attacks.
https://www.owasp.org/index.php/OWASP_HTTP_Post_Tool

I will be using that tool on my site to make sure the iptables are working correctly. I know iptables can't really be a full blown ddos protection/ping ddos/etc. But it can certainly help against script kiddies and programs like that?

Thank you,

Maounique

If you are ovz it may not work as maybe not all modules there, however, I feel like your count is too high for 10 seconds. You need to blacklist the IP if it exhibits that behaviour. Even if it works, your server will still die, I think, with that kind of rate.
M

Nexus

Yeah, I tried to modprobe the module.

"Module xt_state not found."

Is there a way for me to install this module? I am not to keen on messing with kernal data, nor do not want to mess anything up. I am indeed on openVZ btw. OPENVZ support's xt_state right? (As you can tell, not to keen on linux I apologize.)

Right now, I am using basic nginx limit connections / requests module/etc, which is working fine. But I need to use media layer's ddos shell script and iptables.

I mean there's only so much you can do to protect yourself on the end-user vps side, the host hopefully will null route incase of a really bad one? ^_^

craigb

Those iptables rules look solid but as others pointed out, your Linux kernel doesn't support that feature. First hit on GOOG for modsecurity and dos gives you a configuration to use/tweak: http://blog.cherouvim.com/simple-dos-protection-with-mod_security/.

Nexus

Sadly I am with NGINX @craigb

I am using limit_conn perip 3; but, when I use this tool:

https://www.owasp.org/index.php/OWASP_HTTP_Post_Tool

I still can take down my site. I am also using this module for nginx:

limit_req_zone $binary_remote_addr zone=pw:5m rate=1r/s;
limit_req zone=pw burst=40 nodelay;

Seems like these do not work with http flooding? Doesn't seem right, one of the nginx developers even said here: http://forum.nginx.org/read.php?2,212922,212927#msg-212927 That the limit_conn_zone should work.

If anyone wants to take a stab at my conf files to see what I am doing wrong, please do so. Zen said he'll look at it l8er, very appreciated

Here they are:

This is the nginx.cnf

http://pastebin.com/duhpJGrU

And this is the php server/location
http://pastebin.com/wB33Gv6u

That http flooder tool still will take down my site, something doesn't seem right, I am using all available methods to prevent this in NGINX.. (Well so I think?)

craigb

@nexus sorry, you even wrote that in your original post...

I don't know nginx but your configs match the docs. Have you tried setting Limit_conn_log_level to see if your connection limiting is triggered or not in the logs?

Nexus

@craigb, nice idea.

Cleaned logs, did the http tool for 5 seconds, filled my log's up with 16mb of data. lol

Here is some of it:

http://pastebin.com/uAeXKY1c

Doesn't look like my modules are working?

Hmm, I am totally stumped.

craigb

Is there an error log with nginx? I checked the source code for that module (https://github.com/phusion/nginx/blob/master/src/http/modules/ngx_http_limit_conn_module.c) and when it applies connection limits it should return an http 503 to the client (whereas your logs show 400 comsistently) and logs a message saying it applied limiting (string is "limiting connections by zone").

Nexus

@Cragb, for my error.log I receieve.

2012/09/03 02:48:10 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:10 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:11 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:11 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:12 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:12 [crit] 1753#0: accept4() failed (24: Too many open files)
2012/09/03 02:48:13 [crit] 1753#0: accept4() failed (24: Too many open files)

About 21kb worth more.

This is so weird.

When I connect through my site with browsers, with 2 connections refreshing simultaneously to sustain a 2 connections, I do get 503 Service Temporarily Unavailable. But when I run that http tool, it generates a 400 like you said.

It seems like that http tool isn't actually making a simultaneous connection, it's just refreshing on the same ip. This module is completely useless for this type of attack. Looks like I need to go with something similar to http://deflate.medialayer.com/?

craigb

Try setting client_body_timeout in your config file to something short like 6 seconds (amount of time to wait for post data) - you'll probably want to experiment with this number to find a decent trade off (don't want to prematurely cutoff slow mobile clients mid-post). Also increase number of open files available to the nginx worker: http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/.

Need to crash now, will check back tomorrow.

kro

Looks like running out of resources. Look @ ulimit

Nikki

Your problem is that they are opening x connections at a time, exhausting the max number of open files in the kernel.

Your best bet would be to use some kind of in-memory cache if you could, or just mess with iptables/block the IPs being used.

Not sure if this would work, but try some kind of caching module for nginx/varnish cache?

Maounique

I think logging is a bad idea, I mean is fine to see what they are getting, but, in the long term logging alone will DoS your machine and also use a lot of io so the host might suspend you.
I will try a 2 way solution: since they use to request same page, use caching, for one way to reduce load during the attack, second limit connections but much more than what those iptables do. I am not familiar with nginx, but it must be a way to limit connections from same IP and even ban it.
I will go with max one connection per second and ban if it exceeds say 3 a second.
Your situation might require other tresholds, analyse the attack and try to come with the best solution to block it, but not the legitimate ppl.
M

craigb

hey @nexus - you ever get this sorted? what was the solution? edumicate us

Nexus

@craigb

I ran into all sorts of problems with "No table name/etc/etc" when trying to add iptables limit request zones and such.

It was my fault.. I should have known that my host has to enable them on my node.

So I did some basic iptables:

iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 100 -j REJECT --reject-with tcp-reset

iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT
iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 50 -j ACCEPT

and I used nginx to limit connections per ip to 3 and some other stuff if a user bursts more than 50times, he will get a 503 message. (Kids that like to refresh the pages/etc/etc).

@seikan His solution worked well too. My main problem was the xt_state module wasn't even loaded on my node so I kept googling that error for days until I like I said, I finally figured out that I had to ask my host to enable them, which they did

I am using dos deflate which works awesome! I am now just trying to see if there is anything else I can do to help stop script kiddies running crap. I know I can only do so much, but hey atleast I can help mitigate the small dos attacks the best I can right? I know I am still vulnerable for full blown out DDOS Attacks... but meh, I tried.

(I use more iptables from here as well: http://www.cyberciti.biz/tips/howto-limit-linux-syn-attacks.html) I am not a guru at all with linux, but I am learning fast and I love it.

The thing I find weird is some openVZ hosts do not have these modules enabled?

"xt_connlimit
xt_conntrack
xt_state"

These are like the basic ones to help mitigate small script kiddies? I find it pretty weird how most hosts don't even have these enabled by default on openVZ? I mean if anyone is serious about their website they should have some type of DOS protection even if it's not 100% protection... I mean the vps-end user should atleast respect there hosts server and use iptables... IMO.

craigb

@nexus thanks for the write-up...glad you got it worked out