New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hunting Perl Bots with Grep
MTUser2012
Member
I think one of my servers may have a perl bot on it somewhere; one of my WP sites was hacked.
I found this great resource explaining how to use Grep to look for perl bots:
http://pentestlab.org/hunting-malicious-perl-irc-bots/#!prettyPhoto
I'd like to try this approach but the commands don't work on my Centos box:
[root@server ~]# grep -RPn “(system|shell|tcp) *(” /var/www
-bash: syntax error near unexpected token `('
Are there any Linux wizs here who can spot the error? Diagnosing it is beyond my limited skills.
Comments
Try escaping the ()'s with \ so...
grep -RPn "(system|shell|tcp) *(" /var/www
That didn't work as planned, \'s were removed somehow. Basically add \ before each ( and )
Not before each, add \ only before the last (
() are special characters to grep. You need to escape it with a backslash or something.
@udk \\ works?
Thanks for all the help. I am working now looking using Zen's code snippet.
Also netstat -antup, look for suspicious connections, then lsof -p PID to find the path to process/script/binary. Also lsof -f is a good one.
Nothing in the home directory. Anyone have any suggestions on where the hackers usually put these files?
I am also running rootkithunter and clamav, but nothing so far.
There has to be one somewhere. I have 4 processes with unknown users running perl with fake referrers that are 60 to 75% of the server load.
@MTUser2012 this might give you an idea:
ls -l /proc/PID/cwd
ls -l /proc/PID/exe
ls -l /proc/PID/fd
/tmp and /dev/shm quite a bit..
/tmp/.../malicious_file_or_folder
is very common (they try to trick you with the 3 dots)
@rds100. Thank you.
Here is the output from one of the PIDs:
[root@server ~]# ls -l /proc/9908/cwd
lrwxrwxrwx 1 516 513 0 May 1 12:09 /proc/9908/cwd -> /
[root@server ~]# ls -l /proc/9908/exe
lrwxrwxrwx 1 516 513 0 May 1 04:10 /proc/9908/exe -> /usr/bin/perl
[root@server ~]# ls -l /proc/9908/fd
total 0
lrwx------ 1 516 513 64 May 1 11:54 0 -> socket:[3815025094]
l-wx------ 1 516 513 64 May 1 11:54 1 -> pipe:[3820394696]
l-wx------ 1 516 513 64 May 1 11:54 2 -> /var/log/httpd/error_log-20130421
lrwx------ 1 516 513 64 May 1 11:54 3 -> socket:[3819816447]
lrwx------ 1 516 513 64 May 1 11:54 4 -> socket:[854172262]
lr-x------ 1 516 513 64 May 1 11:54 82 -> pipe:[2805043108]
l-wx------ 1 516 513 64 May 1 11:54 85 -> pipe:[2805043110]
[root@server ~]#
Should I run those commands with all the unknown PIDs? Is there anything here besides this thing is in /usr/bin/perl that I could use?
@MTUser2012 there is also cat /proc/PID/cmdline but that can be overwritten by the process.
Just dig in /proc/PID/* and see if you find any clues about these processes.
Did you try lsof -p PID ?
@rds100 & @twain
Thanks for the help. I found the tmp/... folder and deleted it. I could see the running processes pointing to it. I couldn't kill them, they just kept popping back up so I rebooted and all the bad stuff appears to be gone.
clamav found a bad file in a WP theme cache that it deleted so I am guessing that was the back door into the site.
Now update WP and the plugins
I would not be so sure... If it is possible just re-image the node. and most importantly find and fix the source of the vulnerability.
I have dealt with a few of these things, they are very tricky to find and fix.. I have always re-built the nodes whenever they are compromised.. If you do not fix it properly, your node will become a part of a botnet...
@ATHK & @vijayrajah
The problems all seems associated with some WP sites that I bought from one seller. These sites were all updated, but appear to have been hiding files that were there when I bought them. My mistake, I think, was assuming that the these sites would not contain something bad. What I have learned is to check the VPS regularly with ClamAV, and the root kit checkers, and to use top to look for unidentified processes. Numbered not named users are a problem.
I used these tools on my other VPSs and found a few things, again from sites that I bought, that are now gone.
When I build sites myself, I know that they are clean. But what about sites that I buy? What I would like to be able to do is to scan site files before I put them on the server to look for exploits, I am not sure how to do this. Can someone help me with the best way to check code before it is on a server?
Can you not buy from this seller again?.. Give him some feedback etc...
Also, I have seen some scripts, that modify system utilities (like ps, top etc...) hence they are not visible when you execute those commands.. you should check the contents of /proc/ filesystems. It contains all of the PID's of all the processes that are currently in the system. ( as @rds100 mentioned in this thread)
I'm not sure about efficacy of ClamAV never used it myself.. have heard good things about it.. But still, there may be a few (0day ?) vulnerabilities that it can miss
IF there is a bot/trojan/script that is running as root.. I would never ever again trust the system to be clean, no matter what scans i run. Granted this is being paranoid.. But the risk Vs benefit does not justify running a un-trusted server...
Take regular backups (atleast daily).. and check the backups.. Check for things like scripts that start with a '.' and are owned by root...
Never run your webserver as root.. Do a chroot if you can.. I run my apache as 'apache' user and bind to a higher numbered port. I do packet forwarding using iptables.. (if packet reaches port 80, send it port , using nat tables and PRE-ROUTING chain). This will help a little.. make sure that webserver's user's login shell is set to /sbin/nologin... Every little bit helps...
keep an eye on /tmp and /dev/shm and /root directories (as @twain mentioned here). Especially for file that start with a leading '.', or leading blank space.. or some other funny looking file names... (non-printable characters...)
Buy from Themeforest not some cheap ass Asian of Russian site..
@ATHK, you don't buy sites from Themeforest, you buy themes. The sites I bought came from a US seller with a good reputation on Digital Point.
But after working on this for a long time, I finally figured out the problem, timthumb vulnerability. This is sad on two levels: 1. In 2012, a seller is selling sites with a vulnerability discovered in 2011 that are not fixed; 2. I am stupid enough to think that you wouldn't sell someone a site with a back door, so I don't do my own due diligence checking for this vulnerability. Lesson learned the hard way. Sigh.
Unfortunately, s**t happens -- glad you figured it out.