New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@BradND Have you found anything regarding possible attack vector? I did some rev-eng last night on the lib you gave me and came up with nothing outside what has already been discovered in the thread at WHT :S
+50 pages of the thread at WHT and honestly, is full of bs... Most posts are useless, and the few useful ones are being ignored.
Like one inclined to a flash infection in our machines stealing credentials. That seems a good possibility to something not related to SSH.
@jack problem? Did it specifically mention nodedeploy at all?
I am wondering if the SSH implementations with 2-step auth are vulnerable too.
In the meanwhile, probably I will be implementing port knocking, because restricting by IP isn't the best option in MX.
It's hard to freak out over it with so many wild assumptions about where it's coming from.
That's why you should really freak out over it. Not knowing for sure how it got in the first place. Without knowing that, anything you do to mitigate would be pointless since it could very well happen again as some folks over at WHT have posted.
I just find it hard to believe that they're getting root access without setting off any flags. It's tough because while I'd trust a few people here, I'd trust most people at WHT to lie and claim they had their servers secured when they didn't. That whole forum is full of people who like to pat themselves on the back.
Payload isn't known as yet, that's the problem. It's all very circumstantial with no real common denominator. I've seen it reoccur on a server however no extra file was added on either occasion.
It can't be based on key loggers either, at least not in our case, for various reasons. It must be an issue server side and from what is being said it must be ssh related.
So far I've not found a compromised machine, but I'm fairly confident in my alerts that someone gaining access without me knowing is a sign that they either have an incredible exploit or they're just plain brilliant. That to say that if I find myself compromised later, I'll be pretty darn scared. Wouldn't mind having a container of mine compromised though, so I could join in the fun.
That's part of it, they don't leave any obvious trace at all. Unless you have pretty extensive logging and alerts you won't necessarily pick up on it through normal daily routines.
@Jarland Doesn't appear to have affected any of our centos machines (ND stuff) but my work (debian stuff was) so it's weird indeed.
Baffled, mildly fired up, nothing to over analyze. It's the chase I live for...
So, if I don't see libkeyutils.so.1.9, I'm in the clear?
Well, from the "first version".
Probably the creator will do modifications to hide it better.
I know some of you guys are following WHT a bit more closely than me on this. Looks like strong indication of a keylogger being the root cause? It's certainly a favorable outcome given the events.
so far nothing suspicious on my VPSes (all debian), although I got 1 machine with libkeyutil1.4 and 2 different md5hash of libkeyutil1.3
the different between those libkeyutil1.3 is the OS template that I use, the one that has libkeyutil 1.4 coming from my stacklet (provided by my vps provider)
on my openVZ minimal template (taken from openVZ website) is 868d4a3cd978a29e7cde097c99884db7 /lib/libkeyutils.so.1.3
where the official should be:
6affec20aec2654d8906573e2495f289 libkeyutils.so.1.3
I think one of big "hole" in the VPS system is the safety of os template it self, many time we 'assume' it is safe. On provider that allow us to upload our own iso, that would be fine (KVM based mostly), but most of openVZ provider is not.
this also remind me about OVH template, where in my opinion very customize for their need / purpose or on the other word it just non-standard.
one scenario, if attacker got access to node server, he can upload his custom template (centos will be the first choice), replacing the existing one, and just wait until his template got deployed.
better double-check your templates.
Well, latest infos show that this leads to a local vuln.
And something I've found...
http://www.linuxquestions.org/questions/blog/unspawn-2450/simple-clamav-sig-for-lib64-libkeyutils-so-1-9-contents-35316/
This moves you to this one https://access.redhat.com/security/cve/CVE-2013-0871
So, the question is, someone who uses his server in a private way, has been compromised?
@yomero this isn't for sure yet. This was actually mentioned very early like this is possible a local root exploit, which happens from time to time.
But for this you still need to get a local shell in first place. And this is the most interesting part, the attack vector. I.e. how do they "get in".
There are speculations on this. Two of the most common theories I saw are locally stolen passwords or web php scripts exploited. Yet unknown for sure.
EDIT: I've found a pair of my physical CentOS boxes which had ssh open from the Internet (mostly I've restrict ssh to my subnets only with iptables), but they didn't had webserver. And they weren't compromised (at least I didn't found weird libkeyutils libs and network connections).
I didn't say this was sure...
Random, but Found that IP that I saw in the logs trying to get in past infection on blocklist.de's site for trying to login / attempt logins via SSH on thier servers.
I'm beginning to think this may be SSH daemon related, as they were trying to probably exploit a Honeypot SSH server.
http://www.blocklist.de/en/view.html?ip=46.105.20.166
So the thing is making its way around / scanning IPs and attempting maybe an overflow / remote code execution on sshd?
Theory : Scanning bot on a VPS at OVH looking to exploit SSH buffer overflows / a remote exploit.
If it gets in, creates a randomly generated file name i.e: rfow993 which is moved and in some instances chattr'd to the file name(s) seen /lib64/libkeyutils.so.1.9, moving it from /home/tmpp.
This is why my immunization idea of touch, chmod and chattr has worked, it's spreading itself automated via a script probably running 24/7.
Thing leaves quite quickly, knowing machine is rooted and now a part of their arsenal.
-2/20 11:30 PM MST (For when egilette on WHT finds this and decides to take credit)
Initial attack vector still isn't know but more about sshd rootkit can be seen here http://isc.sans.edu/diary/SSHD+rootkit+in+the+wild/15229
This may or may not be useful:
I ran into a VDS that was exploited using this (I believe). You would login and see all the sleep 7200 and such running. I had one of our Development Operations guys who clean up rootkits and servers in their sleep take a look into this, and they were even at a bit of a loss. What we did notice while checking into it is that it seemed they were gaining access through a challenge based authentication schemes. We would notice that 'ChallengeResponseAuthentication no' was set to yes in the /etc/ssh/sshd_config file. We were not at the time able to find the way they were able to gain access using this, however the /var/log/secure would show they root was logging in with the challenge authentication.
It seemed as if they were using a rootkit of types because every time they would gain access they would upload a payload which would roll out a spam script which would immediately start spamming. Once they would upload and start the spam script, they would remove the script and leave it running in memory. They would also setup a site in a custom nginx server which they would use to collect statistics from their spam (they would include a link back to the server in the spam). They deployed the nginx server in the same way, removing the configuration file and executable as soon as it was executed. If you would go to the link your self it would redirect you to Google analytics web site, but at the same time the link would include your e-mail address so they would collect working e-mail addresses before redirecting you.
In the end, as the attacker kept regaining access, we removed the sshd package and installed the most current one on the server. This finally removed their access to the server, so it is for sure an sshd issue. It was also a CentOS 5.x server.
I hope this helps with troubleshooting the issue if you think you have found a server exploited. As I said though, this could just be a variation of a very tricky root kit, as we were never able to confirm for sure how the server was originally exploited (Access could have been gained through a vulnerable service first, for all we know, after finding the server in the state it was, all services were upgraded).
My 2 cents.
Cheers!
From cPanel:
You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.
As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
Gets worse:
From Steve at Rack911's work : http://www.webhostingtalk.com/showpost.php?p=8569892&postcount=1185
If your server's infected, and you utilize it to SSH to another server, that password's sent off in plaintext.
This is going to be bad. I.E: data center core servers / employee VPN servers and the way it's spreading.
Started with cPanel, and possibly other vectors - but from this: We know that any one who contacted cPanel was rooted due to their servers being compromised (and sending off the details of your passwords to the attackers from their machines) - and the same with your machine. Rinse and repeat except your server's infected and sends out credentials when you utilize SSH on it to access remote machines now.
I've been following this issue since it came up seems that Redhat has fixed the issue according to this report - http://secunia.com/advisories/52312/
https://rhn.redhat.com/errata/RHSA-2013-0519.html
All the talk about it is pretty limited as everyone is just crapping themselves .. hard to find a straight answer...
Local user escalation, hmm. Glad virtpanel's java console couldn't properly create a user account if its life depended on it. A victory nested in a failure.
i'm lucky my cpanel server was not (yet) attacked. they say blocking your SSH port can prevent this but I can't whitelist my IP since it's dynamic. do you think whitelisting a VPN IP would help?
Yes
Make it two VPNs in case one goes down.
That's the problem =/
I prefer the port knocking stuff.... BUT, it sucks when you are using the SFTP service for editing files and so. If anyones has any idea on how to achieve web development + port knocking share it :P
IPMI + no listening ports besides web server
I wish I could pull that off and still operate as desired