How to detect source of "attack"?

Amitz

Dear all,

I run a gallery website on a KVM VPS. Generally, everything works ABSOLUTELY smooth, average daily server load is <0.6, RAM usage is way under <512MB and bandwidth usage is at an average of 5 mbps.

The website has around 10,000 visitors and 300,000 pageviews each day. It runs nginx, php-fpm and apc to serve its content. OS is CentOS 5.8.

Then, 2-3 times a week (and not following any pattern), server load shoots up and cpu usage does the same.

There is no correlation to online visitors, doing backups, I/O Wait or something comparable when it happens.
'top -c' at that time shows me the most cpu is eaten by php-fpm.

BUT: One correlation to the problems is that during the high loads, bandwidth usage SHOOTS up to 20 mbps and more. Again - There are no remote backups running at that time.

Please ignore the big spike downwards - that comes from a reboot I did.

I have no explanation for these bandwidth usage explosions and am now assuming that there is some kind of attack running against the server. I really need your help at this point! What can I do to track down the source or cause of these bandwidth usage spikes? Is there any command that I could use to see what IP from the outside is using which amount of bandwidth in realtime? I am really at my wit's end and sincerely hope that someone of you can push me in the right direction!

Thank you very much in advance and please excuse my humble english. I am not a native speaker...

Kind regards
Amitz

rds100

What do the httpd logs say during the spikes, any unusual high activity? Many requests from the same IP? Could be some sladshot-like effect if one of your images gets too popular.

DanielM

netstat ?

Corey

Looks like you are probably getting crawled by a search engine bot. Check your access logs?

efball

If its a crawler try adding this to your robots.txt

Crawl-delay: 10

netomx

Reading this,a nd reading my error logs, I found this:

[Sun Jun 17 12:45:12 2012] [error] [client 74.63.216.123] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

lol someone is port scanning there

Amitz

Thank you, guys!

I am quite sure that some visitors try to crawl the whole site over and over again with website copiers. I already try to bock them using the following direction in my nginx configuration:

   if ($http_user_agent ~* (HTTrack|HTMLParser|libcurl|discobot|Exabot|Casper|kmccrew|plaNETWORK|RPT-HTTPClient)) {
   rewrite ^ http://leecher.wodrun.com/leecher.html? permanent;
#   return 403;
   }

Unfortunately, I have no logs that I could check, because I have turned them off. The website is so busy that the logs grew to more that 7 GB per day and I do not have that much disk space available.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

shows me that there is no single IP with more than 60 simultaneous connections.

Google Analytics shows no increase of traffic from sites like reddit or slashdot.

So there is no way to check things in realtime?

Again, thank you very much for your help!
Amitz

netomx

@Amitz said: I am quite sure that some visitors try to crawl the whole site over and over again with website copiers. I already try to bock them using the following direction in my nginx configuration:

this doesnt stop them, they can easily change it to Mozilla and you're done. Also, with wget, you can limit the connections. Maybe you can post a hidden link to check what IP is trying to get that link? (a bot, for example)

joepie91

@Amitz said: shows me that there is no single IP with more than 60 simultaneous connections.

60 simultaneous connections is fairly significant. How many IPs have, say, >20 connections?

Additionally, it's probably a good idea to turn on access logs temporarily around the time the attack would normally start, to see if there's anything abnormal. After that, you can turn them off again.

Phil

Did you try limiting simultaneous connections and bandwidth with nginx configuration ? Something like:

# Limit connections to 8 per IP
limit_zone evasive $binary_remote_addr 4m;
limit_conn evasive 8;

# Limit rate to 128 KB/s after 5 minutes
limit_rate_after 5m;
limit_rate 128k;

Amitz

@netomx said: this doesnt stop them, they can easily change it to Mozilla and you're done. Also, with wget, you can limit the connections. Maybe you can post a hidden link to check what IP is trying to get that link? (a bot, for example)

Yep, I know. I can only block the total noobs like this. Webcopiers can be a pain in the neck for anyone running an image gallery ...

@joepie91 said: 60 simultaneous connections is fairly significant. How many IPs have, say, >20 connections?

About 10-15% of all IP have more than 20 connections all day long. Most of them in TIME_WAIT state.

@joepie91 said: Additionally, it's probably a good idea to turn on access logs temporarily around the time the attack would normally start, to see if there's anything abnormal. After that, you can turn them off again.

I agree. The main problem is, that I never know when it will start. It follows no pattern. Everything can be quiet for days, then it happens 2-3 days a week, always at different times. And the whole thing is mostly over after 1-2 hours. Most of the time, I just see it afterwards when looking at the munin statistics. I will now configure CSF/LFD to send me an eMail when the load increases to >2 and to let it automatically execute a script that turns on access logs. Hopefully I can read the information from it that I need...

@Phil said: Did you try limiting simultaneous connections and bandwidth with nginx configuration ?

Thank you - No. I did not. As it is an adult image gallery, visitors tend to open many tabs in their browser at once to... well, have their hands free, I guess. Therefore I did not want to limit them to much. But it seems that it would be a smart move to do so. Do you think that 8 connections per IP are enough not the bother the classic Average Joe visitor?

Amitz

Ah, to bring one question up again:
There is indeed no command line tool that is able to show the bandwidth rate that IP X is receiving from the server? Something like vnstat live information with direct correlation to connected IPs? I imagine an output like:

2 IPs connected to the server:
xxx.xxx.xxx.xxx using xxx mbps
xxx.xxx.xxx.xxx using xxx mbps
.
.
.

RoboCot

What you need is iftop.

And now you owe me your dirty address he he he

(slurp!)

Phil

@Amitz said: Do you think that 8 connections per IP are enough not the bother the classic Average Joe visitor?

If I remember well, most recent browsers allow up to 15 as a default. So you may try with 15 first to see if it helps (better than 60 or more from downloaders), and lower if needed. Since setting is usually per server, number of tabs should not matter.