All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What does this mean? p0wned?
I just received this email with the subject line "Privileged Account logon used" from my server running on a kimsufi.. The server has a standard LAMP with webmin running on it. It has some home brewed php and perl scripts. There's a utorrent server, and yes, a vncserver for a non privileged user.The web scripts are unaudited and probably have vulnerabilities to unescaped code. Does it mean the server has been breached? It's the first time I received this mail, and now I got around 6 of them. I immediately logged in and shut it down.
XDG_SESSION_ID=c3914 TERM=xterm SHELL=/bin/bash SSH_CLIENT=122.174.195.243 14693 22 SSH_TTY=/dev/pts/0 USER=root MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/root LANG=en_GB.UTF-8 SHLVL=1 HOME=/root LANGUAGE=en_GB:en LOGNAME=root SSH_CONNECTION=122.174.195.243 14693 myserverip 22 XDG_RUNTIME_DIR=/run/user/0 _=/usr/bin/env
122.174.195.243 seems to belong to a Access Business Group,DSL Services 101,Chennai, India, whom I've never heard of.
What's this? The server has ssh enabled for root with a complex password which even I dont know. I usually login with a private key.
Comments
You have been hacked.
So.. basically collect /var/log and reinstall? Would they be informative as to where the hack happened from?
It's unmanaged so it wouldn't be realistic to trouble anyone with finding out for you why it happened, but yes, looking at the logs might help you find out and learn something.
And reinstall, or just clean up if you're confident you know how it happened and can reverse any changes made.
Is it safe to login as root, disable password access, webservers, disable users with shell access, and work on it a while to collect logs and backup? Maybe any scripts to do this quickly? I know the backup contain compromised data, but the scripts are written by me, so I could find out anything that stands out. Plus they've been pushed to git so could do comparison.
Sounds like a plan. As long as nothing bad is happening you can check it out. Use
who,topnetstatetcroot login permitted plus running X ... what could possibly go wrong? (I guess, you got an answer now).
As for your question, let me put it differently: What are you afraid to risk or to loose when connecting again to a pwned server? So, yes, go there and grab the logs (but be sure to understand that logs on a pwned server should be looked at mistrustingly).
..and its running systemd. Of course.
Yea, this server was a calculated risk.. I needed X to batch download (a site using complex javascript) from a browser window that wasnt working in headless mode.
Thought the vulnerability was patched? Sorry, I'm a hobbyist. Still in learning phase regarding server security.
It was just a drive-by snipe at systemd. Seeing that your user messages have env vars with PAM auth through systemd set, I wouldn't doubt it was somehow involved- but I can't say how they got in.
Should I be retaining ovh user? It seems to have shell access.
Just logged in recovery mode via Netboot.
It's time for a complete wipe. If you're planning on getting whatever possible information- just realize that your logs (anything which isn't logged through a remote syslog) are compromised, too. If the whole filesystem wasn't too much, I'd make a snapshot, then blow it out and restart.
Of course, I'd start by updating everything after reinstall, then checking your scripts for issues. Leave X11 off this time.
hmm... TODO:
Pull backupsdone and remote syslogUse ChromeDriver with xvfb
I believe that OVH user is for the kimsufi stats and stuff you see in the Kimsufi control panel.
Does anyone know what caused the server to send this mail to me? AFAIK I didnt setup anything like this on this server..
Edit: Oh yea..I'm reinstalling. Seems the compromise occured almost a month ago..if this is to believed, and this is the right ip..
at least you should firewall lock the ssh port for one ip (your home static ip or some vpn etc ...)
i learned myself 
also you could boot in rescue mode and check modified binaries at /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, i bet some of them was replaced
had the same scenario 
Unfortunately, over here, we dont have the advantage of static ips.. maybe a vpn..
But if root is the only user with ssh access, and password is disabled, the chance of access through ssh is neglibible, isnt it?
Hmm.. Compare them to some kind of database? Can clamav do this?
What I've done so far..
So... From rescue, edited passwd file, deleted vncuser, removed shell access for all except root, recursively deleted all keys, added a new key for root, disabled passwd login for all ssh users, deleted vncserver, removed apache2 server. Reinstalled the linux kernel (dont know if that was required), cleared crontabs.
Edit: I guess rootkits can still wreak havoc
Edit2: Discovered chrootkit and metasploit
Edit3: chrootkit says no rootkits have been installed. Can I take a moment to give a sigh of relief? Installed tripwire. Logging w.. Running w in a shell..
no matter what, i DONT trust anyone who can REACH ssh port so thats why i am locking it to only known IP (i can sleep better then
)
if you do ** cd /usr/sbin** you can see changed binaries if any, but you need the specify date of modify/install (if you did apt-upgrade or yum etc ...), so i assume anything thats have newer modify or it is unknown file is high risk probably
The hacker still can have some backdoor anywhere well hidden, once i remember i had hacked version of ssh, which all what i typed into that hacked version of ssh was streamed to fkin hacker.
backup, check if anything is changed in your scripts, and do a CLEAN INSTALL OF SYSTEM to be 100% sure your system is clean.
I see your point, but it's not entirely true. That's like saying your house is compromised because someone can walk up to the front door. Accessing the port does nothing in and of itself. It's like accessing the login page of any online service.
Disabling password logins and using keys is pretty good in itself. You can also use a non standard port to reduce log entries. If you have the benefit of static IPs you can then limit access to those, but a lot of home users don't have that benefit (using multiple VPNs in different locations can make up for a lack of static IP access).
No.
cdinto the directory is not going to tell you anything. File time stamps can be modified just like logs can be. If you're not going to diff them against known good ones or run hashes on them, don't waste your time with this.If you cannot find the entry point you need to find someone to help you find it. Once you've found it you can patch and do a clean install.
chrootkit should do this right?
Maybe... but there's nothing stopping a targeted attack from doing things outside its scope. As with all these things.. we just have to exercise diligence.
Patches md5sum and friends
Well, so much for your calculation ...
No, the systemd vulnerability is still running. But don't you worry, soon X, ssh, and firefox will be in systemd, too.