Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


High load some times - lots of tcp connections - why?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

High load some times - lots of tcp connections - why?

myhkenmyhken Member

I monitor my server loads on all server at all time. Several times per week I see my load goes up to 1.0 - maybe 1.5 and stay high for quite some time.

I have start using:
netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1
to check how many connections is active on my servers, and when the load is high, the numbers of connections is high.

I then run:
netstat -an | grep :80 | sort
to find which IPs has most connections, and almost every time I find a IP with a high numbers of connections, like this (this is not a high number, but it caused a load of 0.20 for 30 minutes)

I then use http://whois.domaintools.com/ to check the IP. And it always is in different countries. But since most of my pages is in Norwegian, it's easy to understand that a IP from etc Australia with lots of tcp connections is the issue.

My solution is to add the IP to my firewall deny list, and then the load goes back to normal.

Can I automate this process so I can stop this whenever I don't are online?

And what are they doing on my server? When the load is high, almost no bandwidth is used.
It's only the server load that is high.

Comments

  • rds100rds100 Member

    Look at the web server logs? See what requests were there by this IP.

  • myhkenmyhken Member

    @rds100 said:
    Look at the web server logs? See what requests were there by this IP.

    Searching my logs gives me nothing.
    I have searched all logs in this picture, is there other logs I should look in?

  • rds100rds100 Member

    I hope you search just for the IP (2.250.35.236), not with this ffff nonsense.

  • myhkenmyhken Member

    I searched for (203.29.27.10) that was the IP that caused the high load. Did not search with the ffff only the IP

    I get this from every logfile:

  • rds100rds100 Member

    Then this is not the right logfile or you don't have the logging setup correctly it seems.

  • myhkenmyhken Member

    @rds100 said:
    Then this is not the right logfile or you don't have the logging setup correctly it seems.

    Found the logfile, since I'm using virtualmin, they are stored in /var/log/virtualmin and then access log for each domain.

    It's seems like the user tried to access /wp-admin page:
    203.29.27.10 - - [11/May/2014:18:55:50 +0200] "POST /wp-login.php HTTP/1.0" 200 3598 "-" "-"
    It's lots of entries like this in the log.

    A couple of months ago, I deleted all admin users from all my Wordpress installations, and changed it to a new user. After that I have seen a drop of access on /wp-admin on all sites. But it's seems like still someone try to hack my sites from time to time.

    What is the best way to auto stop to many tries on /wp-admin?

  • rds100rds100 Member

    .htaccess and limit the access to only several IPs (all yours).

  • M66BM66B Veteran

    It looks like your WordPress passwords are being guessed. One of my WordPress sites was attacked for almost two years from the same Russian IP address (!). Make sure you do not use 'admin' as user name, because the password guessing is almost always for this user name. Maybe you can use a WordPress plugin to limit the logins (too). There are several in the wordpress.org repository available.

  • myhkenmyhken Member

    M66B said: Make sure you do not use 'admin' as user name

    Like I said in the post over, I deleted all admin users (user #0) on all my sites (and my customers) and used a different username on the installations.
    And it stopped some huge attacks I had with several thousands tries to login each day.

    But it's seems somebody still try some small attacks then.

  • M66BM66B Veteran

    The Russian IP I spoke about was guessing a password every 15 minutes, so there are also slow attacks in an attempt to cover the attack.

  • rds100rds100 Member
    edited May 2014

    Some pesky bots were very persistent in trying to open /administrator/index.php on one of our sites. There was no such file there mind you, but they kept trying and trying. So what did i do? I created that index.php they so much wanted:


    <? header('Location: http://proof.ovh.net/files/10Gio.dat'); ?>

    Of course first i tested and verified that these bots were stupid enough to follow the redirect.
    Let's waste some bot bandwidth :)

  • myhkenmyhken Member

    lol

  • MaouniqueMaounique Host Rep, Veteran

    Lol, that is cool :P

  • myhkenmyhken Member

    @rds100 said:
    .htaccess and limit the access to only several IPs (all yours).

    So simple, just applied .htaccess file with my current IPs, and thats it. Nobody else can logon wp-admin.
    Luckily that no of my customers need access to wp-admin since I do all updates.

  • RazzaRazza Member
    edited May 2014

    @rds100 said:


    <? header('Location: http://proof.ovh.net/files/10Gio.dat'); ?>

    i might try that trick too, as some bots keep on hiting a non existing phpmyadmin on my site

  • jarjar Patron Provider, Top Host, Veteran

    @rds100 said:
    Some pesky bots were very persistent in trying to open /administrator/index.php on one of our sites. There was no such file there mind you, but they kept trying and trying. So what did i do? I created that index.php they so much wanted:


    <? > header('Location: http://proof.ovh.net/files/10Gio.dat'); > ?>

    Of course first i tested and verified that these bots were stupid enough to follow the redirect.
    Let's waste some bot bandwidth :)

    Well I know what I'm doing on all my servers now. Maybe a little revenge on "BOT for JCE" :)

  • AmitzAmitz Member

    @rds100

    What a nice and simple idea! The malicious bots crawling my site every day will have LOTS of fun with this! :-)

  • geekalotgeekalot Member
    edited May 2014

    @myhken, Use an IDS, such as Fail2ban or LFD (CSF) or Snort, and write a rule specific to your application that blocks the IP's.

    i.e., understand what the "failure" URL/response/HTTP status code is for an unsuccessful attempt and key in on that. Obviously, also pay attention to your own users' mistaken login attempts and permit enough attempts to prevent blocking genuine mistakes.

    If you find enough attacks from a particular subnet, consider permanently firewalling the subnet.

    You have to really study your log files (and know where all the logs are for your setup) to see what the attackers are doing and customize your response for the attacks you are seeing (in addition to responding to the "standard" attacks & threats).

    This can work very, very well + save bandwidth + keep servers performing well (and responsive for legitimate users).

    EDIT: The cute little redirection trick might get YOUR IP's banned or put on a blacklist of some kind by the provider you redirect them to. Better IMHO to nullroute route them then if you are going to do something like this.

  • @myhken, maybe you could also try this plugin (Rename wp-login.php) to rename the default /wp-admin to something else.

    I've changed my default wp-admin, and the attack stopped.

  • BinaryLaneBinaryLane Member, Host Rep

    A lot of our customers are seeing this (brute force login attempts) and some are having performance issues due to the sheer volume of requests. This style of wordpress-specific attack seems to have increased in the last few months.

    The plugin to rename wp-login is what I have been recommending

  • rds100rds100 Member
    edited May 2014

    @Razza @jarland @Amitz first redirect it to a file you host and check in the logs that the file is being downloaded. Then change the redirect to some big file in a big network that wouldn't mind the extra bandwidth wasted.

    First i thought using cachefly's test file, but it's only 100MB. If someone knows of a big download file on a big CDN that wouldn't hurt anyone's bandwidth bill - let me know.

    Thanked by 1jar
  • J1021J1021 Member
    edited May 2014

    rds100 said: First i thought using cachefly's test file, but it's only 100MB. If someone knows of a big download file on a big CDN that wouldn't hurt anyone's bandwidth bill - let me know.

    I doubt OVH will have problems with bandwidth usage.

  • AmitzAmitz Member

    Concerning the main topic: I had exactly the same problem with the Wordpress installations on my servers and this plugin here http://wordpress.org/plugins/bruteprotect/ stopped it all, including the high loads caused by the brute force attacks. Like this, you do not have to block access to wp-login.php and external authorized users can still access it!

    Thanked by 1myhken
  • rds100 said: @Razza @jarland @Amitz first redirect it to a file you host and check in the logs that the file is being downloaded. Then change the redirect to some big file in a big network that wouldn't mind the extra bandwidth wasted.

    First i thought using cachefly's test file, but it's only 100MB. If someone knows of a big download file on a big CDN that wouldn't hurt anyone's bandwidth bill - let me know.

    OVH host the largest 'do whatever you want with them' test files I know of at up to 10GB. Here you go:

    http://proof.ovh.net/

  • rds100rds100 Member

    AThomasHowe said: OVH host the largest 'do whatever you want with them' test files I know of at up to 10GB. Here you go:

    http://proof.ovh.net/

    Guess what i am using at the moment? Hint, scroll up a little :)

  • tomletomle Member, LIR

    Try fail2ban, you might need to create a custom rule. Or install modsecurity together with fail2ban.

  • rds100 said: Guess what i am using at the moment? Hint, scroll up a little :)

    It might be 12 noon but it's still too early in the morning for me okay? ;)

    What size test file are you looking for? There's a good number of 1GB test files, like I said though OVHs is the largest I know of. ThinkBroadband have a 1GB test file for example on both IPv6 and IPv4.

  • rds100rds100 Member

    @AThomasHowe i was not looking for a test file, but for a large "go download this file to waste your bandwidth" file. I ended up using OVH's 10GB file.

  • rds100 said: @AThomasHowe i was not looking for a test file, but for a large "go download this file to waste your bandwidth" file. I ended up using OVH's 10GB file.

    Well yeah but you're better off using a test file for that ;P

Sign In or Register to comment.