New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
You're talking to /r/iamreallysmart personified, you know. He has timestamps and checksums memorized for every distribution, every release, and every patchlevel. Pleb.
Agreed. However there are cases where it is not feasible full reinstall either.
Hence, remove systemd and pin sysvinit.
I have found the way it breaks into my system.
Firstly, I checked auth.log and didn't found any ssh logins from strange IPs. All successful logins are from my home IP and another remote machine. It makes me confused. Then, I spent some hours to look carefully on the auth.log and I found this:
Dec 24 02:06:36 palmyran-pve-node23 sshd[1361]: Server listening on 0.0.0.0 port 22. Dec 24 02:06:36 palmyran-pve-node23 sshd[1361]: Server listening on :: port 22. Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: Watching system buttons on /dev/input/event1 (Power Button) Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: Watching system buttons on /dev/input/event0 (Power Button) Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: New seat seat0. Dec 24 02:07:02 palmyran-pve-node23 login[1393]: pam_unix(login:session): session opened for user support by LOGIN(uid=0) Dec 24 02:07:02 palmyran-pve-node23 login[1929]: ROOT LOGIN on '/dev/tty1' Dec 24 02:08:41 palmyran-pve-node23 login[1393]: pam_unix(login:session): session closed for user support Dec 24 02:10:01 palmyran-pve-node23 CRON[5282]: pam_unix(cron:session): session opened for user root by (uid=0) Dec 24 02:10:01 palmyran-pve-node23 CRON[5282]: pam_unix(cron:session): session closed for user rootYou can see, there's a ROOT LOGIN from /dev/tty1 at Dec 24 02:07:02. (What the hell???).
After finding the time my server was hacked, I can narrow the search and I found this on syslog:
Dec 24 02:08:41 palmyran-pve-node23 systemd[1]: [email protected] has no holdoff time, scheduling restart. Dec 24 02:08:41 palmyran-pve-node23 systemd[1]: Stopping Getty on tty1... Dec 24 02:08:41 palmyran-pve-node23 systemd[1]: Starting Getty on tty1... Dec 24 02:08:41 palmyran-pve-node23 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument Dec 24 02:08:41 palmyran-pve-node23 systemd[1]: Started Getty on tty1. Dec 24 02:09:01 palmyran-pve-node23 cron[1523]: (*system*) RELOAD (/etc/crontab) Dec 24 02:09:36 palmyran-pve-node23 postfix/pickup[1510]: EDFF25C01C2: uid=0 from=<root> Dec 24 02:09:36 palmyran-pve-node23 postfix/cleanup[1512]: EDFF25C01C2: message-id=<20171224070936.EDFF25C01C2@palmyran-pve-node23> Dec 24 02:09:36 palmyran-pve-node23 postfix/cleanup[1512]: warning: EDFF25C01C2: write queue file: No space left on device Dec 24 02:09:36 palmyran-pve-node23 postfix/pickup[1510]: warning: maildrop/8D51C5C01B8: error writing EDFF25C01C2: queue file write error **Dec 24 02:10:01 palmyran-pve-node23 CRON[5283]: (root) CMD ( /usr/sbin/monero)** Dec 24 02:10:14 palmyran-pve-node23 systemd-timesyncd[613]: interval/delta/delay/jitter/drift 256s/-0.002s/0.036s/0.002s/+20ppm Dec 24 02:10:37 palmyran-pve-node23 postfix/pickup[1510]: 0882A5C01C2: uid=0 from=<root> Dec 24 02:10:37 palmyran-pve-node23 postfix/cleanup[1512]: 0882A5C01C2: message-id=<20171224071037.0882A5C01C2@palmyran-pve-node23> Dec 24 02:10:37 palmyran-pve-node23 postfix/cleanup[1512]: warning: 0882A5C01C2: write queue file: No space left on device Dec 24 02:10:37 palmyran-pve-node23 postfix/pickup[1510]: warning: maildrop/8D51C5C01B8: error writing 0882A5C01C2: queue file write error Dec 24 02:11:37 palmyran-pve-node23 postfix/pickup[1510]: 18E8A5C01C2: uid=0 from=<root>So, 3 minutes from breaking into my server, cron called /usr/sbin/monero. This is the soonest result when I searched this cron on syslog. This monero malware called minerd every few minutes and monitor minerd process.
More (truncated because it's too big):
auth.log: https://pastebin.com/T43t9LCv
syslog: https://pastebin.com/dEvb2AWm
I have not concluded anything yet. But it makes me confuse, really confuse...
Do you use SSH keys?
r o o t k i t
That is why it's showing that /dev/ directory.
No.
I think if the successful login is from ssh, it should be like this:
Dec 16 05:19:28 palmyran-pve-node23 sshd[1710]: Accepted password for root from x.x.x.x port 62015 ssh2
Root kit doesn't always mean it logs in via SSH.
Are you running any services on the machine? Such as Redis, anything like this?
If you can give us some information on how the machine is used, it can narrow it down.
I have seen this happening a lot more recently.
I don't understand why you keep going on about rootkit. I might be a bit out of the loop on this and there sure are different types but if i hear rootkit i pretty much imagine a kernel rootkit. Why would this have to login at all?
Judging by the line about sshd it seems this happend shortly after a reboot? As far as those logs can be trusted it seems someone logged in locally shortly after the machine was booted.
Going to take a look at the logs you posted and see if i happend to notice anything interesting.
Edit: Where does this support user come from? Any logs of it being created? Any more info on this user like |etc|passwd (replace | for /... cloudflare...) entry, user id, groups? Why was the box shutdown and stayed offline for about 5min? Could it be there are some logs missing between shutdown and restart?
Jeez, 23 kh/s is insane.
Just curious and worried about my VPS, does the process shows up when you do TOP?
And how did you noticed the malware in the first place?
This is exactly what I thought. It seems that someone login locally. No log missing between shutdown and restart.
I think the problem is Supermicro IPMI has been compromised. There is another evidence support this. I could not login to IPMI with provided password yesterday and had to ask the provider to reset IPMI.
Yes, it shows on top with about 500% cpu usage (on a 8 core machine). I noticed it by the performance down on the server.
I thought about the same... maybe just a coincidence. but also may be worth asking @virmach to check if that's really a single case here or if someone with access to the customers boxes is trying to scrape off some extra money from his infrastructure.
I am pretty sure he wants to know about possible shit going on.
How do you know? The log just cuts off. No reason for the restart visible at least to me. Can you get more info on this user? Did you install from a template?
I don't have enough experience with IPMI to comment on this.
That's pretty much why i wouldn't really call it rootkit. Any semi serious rootkit would try to hide it's processes.
Anyways, not wanting to point fingers but does anyone else got a Virmach dedi and could check if this support user exists (if the box was installed from a template)?
Yep, the local login in combination with a user called support is really suspicious. That's why i am interested to know if the box was installed from a template. Faking a local login would not make sense for any malware. I mean, seriously, on a rented box it sticks out like a sore thumb.
Here is the ticket I created yesterday to ask about IPMI login issue. At that time, I just want to go to IPMI to reinstall the server.
I thought it may be created by using single user mode when you can access grub boot.
No, I didn't using Virmach dedi template. I installed OS by myself. Moreover, I have some servers with them, all is installed the same way because I ran preseed debian install but only this one is infected.
are the IPMI credentials (user/pw) the same for all of your boxes? maybe check if and what additional users might be enabled in IPMI...
So a smart malware hacker would probably aim for a much lower CPU load with a high chance of not getting noticed at all.
need to run malware scan on the server
he can use clamscan for that
I think it will help to resolve the issue, if not then you can use live CD to boot for clean.
Thanks
Casey K
Skype: live.euboxes
No, IPMI user/pw is different for each server.
If I was him, I will set a smarter procedure for minerd. LOL
Dec 24 02:06:36 palmyran-pve-node23 sshd[1361]: Server listening on 0.0.0.0 port 22.
Dec 24 02:06:36 palmyran-pve-node23 sshd[1361]: Server listening on :: port 22.
Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: Watching system buttons on /dev/input/event1 (Power Button)
Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: Watching system buttons on /dev/input/event0 (Power Button)
Dec 24 02:06:36 palmyran-pve-node23 systemd-logind[1272]: New seat seat0.
Dec 24 02:07:02 palmyran-pve-node23 login[1393]: pam_unix(login:session): session opened for user support by LOGIN(uid=0)
Dec 24 02:07:02 palmyran-pve-node23 login[1929]: ROOT LOGIN on '/dev/tty1'
Dec 24 02:08:41 palmyran-pve-node23 login[1393]: pam_unix(login:session): session closed for user support
Dec 24 02:10:01 palmyran-pve-node23 CRON[5282]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 24 02:10:01 palmyran-pve-node23 CRON[5282]: pam_unix(cron:session): session closed for user root
There has been a string of attacks against SolusVM control panels as well as the VNC services provided by providers. You can tell in this case they likely obtained your control panel credentials through brute force and then using the panel reset your password and logged in as root via VNC (this is how you end up with root logging in on /dev/tty1). Unfortunately providers don't seem to instantiate any type of brute force detection against VNC / SolusVM which allows this to occur if you set a weak password for your SolusVM access and/or VNC access to the server.
My suggestion is that you make sure you set very secure passwords on the SolusVM control panel as well as VNC if you leave it active for your server as well as reinstall things now as someone obviously had full root access to the server other than your self and you have no way to tell how riddled with rootkits and such the server may now be.
I had something similar happen where they managed to guess my VNC password and the attacker just sat trying passwords and rebooting the server to be annoying. I had opened a ticket to the provider to find out why my server kept rebooting only to find that someone was doing so from the console (VNC) (the provider offered no assistance), my only recourse was to disable VNC for the server to prevent further abuse.
My 2 cents.
Cheers!
Thank you for your suggestion. I have 5 servers in the same account of SolusVM but only this server got infected. I guess the problem lies on the IPMI password. I cannot change the IPMI password since Virmach set the user to "operator" only. This makes me even more confuse. If I, with the operator account, cannot change the password of IPMI, how did the attacker change it???
He said it's a dedi so i don't think VNC is the culprit. What describe is a very real scenario though. Adding a second user with uid 0 instead of resetting roots password is probably what any semi smart attacker would do with VNC access so i see how this fits OPs situation.
Also i was pretty much thinking the same thing when i was testing a KVM lately and virtualizor just presented me with an IP and a port when i clicked the VNC button. Like wow, this is insecure. Sure a real VNC client might be nicer than the NoVNC webclient but VNC on its own does not even have encryption and getting access there is more or less instant root. Just send Ctrl+Alt+Del and enjoy single user. I wish i could disable this but it seems Virtualizor won't let me so i just set a very long gibberish password i hope noone will bother bruteforcing. Still leaves a bad feeling.
Edit:
Wait, you actually have SolusVM running on your server and the compromised box is a VM? I don't have a ton experience with SolusVM but if it provides VNC what @TheLinuxBug describes really seems like a good theory on what might have happend. Who knows why other servers were left alone. Maybe their VNC uses a different password or the attacker just scanned for this single port and did not realize the others where there? Last part is admittedly not that plausible. Why wouldn't a successful attacker try different displays? Could have been some automated tool using genorous delays to automate VNC though. Yeah i love speculation
No, sorry I didn't make it clear. I have 5 servers in the same Virmach account (which is running SolusVM) but only this server was infected. The compromised box is a physical dedicated server. The account support who successful logged into the box should be from IPMI.
I seem to recall a semi-common setup for CC resold dedis; I wonder if that information was compromised.
IPMI not being behind a firewall is the problem, If you not using it turn it off once your OS is installed.
I have had several servers from CC in the past with IPMI enabled, all of them were constantly being brute forced. IPMI Enabled front facing servers can also be used for DDOS reflection attacks, had that happened to me.
There is no good reason to have IPMI unless you're behind a firewall
Just to let everyone know: our servers at work have been hacked by the same method. The same mining pool and account is used for XMR payouts! The guy has generated more than 140k $ so far with this hacking!
I have contacted the pool operators (nanopool) and at first they canceled all payouts to the guy, but later resumed them. I have also contacted the provider of the server where he seemingly logged in from (198.12.89.203 hosted by colocrossing.com), but did not get a reply yet. I have also contacted the programmer of the miner software (lukminer), but - of course - he cannot do very much about it.
Greetings!
Nanopool are botnet friendly? Crazy. SupportXMR and MineXMR seem to actually discourage the activity.
Yea, I sent them a message but they eventually resumed the payouts as well.