Hunting Perl Bots with Grep

MTUser2012

I think one of my servers may have a perl bot on it somewhere; one of my WP sites was hacked.

I found this great resource explaining how to use Grep to look for perl bots:

http://pentestlab.org/hunting-malicious-perl-irc-bots/#!prettyPhoto

I'd like to try this approach but the commands don't work on my Centos box:

[root@server ~]# grep -RPn “(system|shell|tcp) *(” /var/www
-bash: syntax error near unexpected token `('

Are there any Linux wizs here who can spot the error? Diagnosing it is beyond my limited skills.

udk

Try escaping the ()'s with \ so...

grep -RPn "(system|shell|tcp) *(" /var/www

udk

That didn't work as planned, \'s were removed somehow. Basically add \ before each ( and )

rds100

Not before each, add \ only before the last (

Rallias

() are special characters to grep. You need to escape it with a backslash or something.

@udk \\ works?

MTUser2012

Thanks for all the help. I am working now looking using Zen's code snippet.

twain

Also netstat -antup, look for suspicious connections, then lsof -p PID to find the path to process/script/binary. Also lsof -f is a good one.

MTUser2012

Nothing in the home directory. Anyone have any suggestions on where the hackers usually put these files?

I am also running rootkithunter and clamav, but nothing so far.

There has to be one somewhere. I have 4 processes with unknown users running perl with fake referrers that are 60 to 75% of the server load.

rds100

@MTUser2012 this might give you an idea:
ls -l /proc/PID/cwd
ls -l /proc/PID/exe
ls -l /proc/PID/fd

twain

/tmp and /dev/shm quite a bit..

/tmp/.../malicious_file_or_folder

is very common (they try to trick you with the 3 dots)

MTUser2012

@rds100. Thank you.

Here is the output from one of the PIDs:

[root@server ~]# ls -l /proc/9908/cwd
lrwxrwxrwx 1 516 513 0 May 1 12:09 /proc/9908/cwd -> /
[root@server ~]# ls -l /proc/9908/exe
lrwxrwxrwx 1 516 513 0 May 1 04:10 /proc/9908/exe -> /usr/bin/perl
[root@server ~]# ls -l /proc/9908/fd
total 0
lrwx------ 1 516 513 64 May 1 11:54 0 -> socket:[3815025094]
l-wx------ 1 516 513 64 May 1 11:54 1 -> pipe:[3820394696]
l-wx------ 1 516 513 64 May 1 11:54 2 -> /var/log/httpd/error_log-20130421
lrwx------ 1 516 513 64 May 1 11:54 3 -> socket:[3819816447]
lrwx------ 1 516 513 64 May 1 11:54 4 -> socket:[854172262]
lr-x------ 1 516 513 64 May 1 11:54 82 -> pipe:[2805043108]
l-wx------ 1 516 513 64 May 1 11:54 85 -> pipe:[2805043110]
[root@server ~]#

Should I run those commands with all the unknown PIDs? Is there anything here besides this thing is in /usr/bin/perl that I could use?

rds100

@MTUser2012 there is also cat /proc/PID/cmdline but that can be overwritten by the process.
Just dig in /proc/PID/* and see if you find any clues about these processes.

twain

Did you try lsof -p PID ?

MTUser2012

@rds100 & @twain

Thanks for the help. I found the tmp/... folder and deleted it. I could see the running processes pointing to it. I couldn't kill them, they just kept popping back up so I rebooted and all the bad stuff appears to be gone.

clamav found a bad file in a WP theme cache that it deleted so I am guessing that was the back door into the site.

ATHK

Now update WP and the plugins

vijayrajah

@MTUser2012 said: clamav found a bad file in a WP theme cache that it deleted so I am guessing that was the back door into the site.

I would not be so sure... If it is possible just re-image the node. and most importantly find and fix the source of the vulnerability.

I have dealt with a few of these things, they are very tricky to find and fix.. I have always re-built the nodes whenever they are compromised.. If you do not fix it properly, your node will become a part of a botnet...

MTUser2012

@ATHK & @vijayrajah

The problems all seems associated with some WP sites that I bought from one seller. These sites were all updated, but appear to have been hiding files that were there when I bought them. My mistake, I think, was assuming that the these sites would not contain something bad. What I have learned is to check the VPS regularly with ClamAV, and the root kit checkers, and to use top to look for unidentified processes. Numbered not named users are a problem.

I used these tools on my other VPSs and found a few things, again from sites that I bought, that are now gone.

When I build sites myself, I know that they are clean. But what about sites that I buy? What I would like to be able to do is to scan site files before I put them on the server to look for exploits, I am not sure how to do this. Can someone help me with the best way to check code before it is on a server?

vijayrajah

@MTUser2012 said: The problems all seems associated with some WP sites that I bought from one seller

Can you not buy from this seller again?.. Give him some feedback etc...

Also, I have seen some scripts, that modify system utilities (like ps, top etc...) hence they are not visible when you execute those commands.. you should check the contents of /proc/ filesystems. It contains all of the PID's of all the processes that are currently in the system. ( as @rds100 mentioned in this thread)

I'm not sure about efficacy of ClamAV never used it myself.. have heard good things about it.. But still, there may be a few (0day ?) vulnerabilities that it can miss

IF there is a bot/trojan/script that is running as root.. I would never ever again trust the system to be clean, no matter what scans i run. Granted this is being paranoid.. But the risk Vs benefit does not justify running a un-trusted server...

Take regular backups (atleast daily).. and check the backups.. Check for things like scripts that start with a '.' and are owned by root...

Never run your webserver as root.. Do a chroot if you can.. I run my apache as 'apache' user and bind to a higher numbered port. I do packet forwarding using iptables.. (if packet reaches port 80, send it port , using nat tables and PRE-ROUTING chain). This will help a little.. make sure that webserver's user's login shell is set to /sbin/nologin... Every little bit helps...

keep an eye on /tmp and /dev/shm and /root directories (as @twain mentioned here). Especially for file that start with a leading '.', or leading blank space.. or some other funny looking file names... (non-printable characters...)

ATHK

Buy from Themeforest not some cheap ass Asian of Russian site..

MTUser2012

@ATHK, you don't buy sites from Themeforest, you buy themes. The sites I bought came from a US seller with a good reputation on Digital Point.

But after working on this for a long time, I finally figured out the problem, timthumb vulnerability. This is sad on two levels: 1. In 2012, a seller is selling sites with a vulnerability discovered in 2011 that are not fixed; 2. I am stupid enough to think that you wouldn't sell someone a site with a back door, so I don't do my own due diligence checking for this vulnerability. Lesson learned the hard way. Sigh.

Microlinux

Unfortunately, s**t happens -- glad you figured it out.