Possible server break-in. What are my options?
Hi there,
I fear that today, after nearly 10 years of taking care of my own servers, I became victim of a server break-in. I don't know how it happened yet, but it seems as if somebody was able login to one of my servers with a Webmin/Virtualmin installation and had a good time deleting random data in the /home directory. It may also be possible that there was some other fault, but I cannot imagine which one. The server was running without issues for more than a year and received all system and software updates on a daily basis. Access via SSH was only possible via SSH keys (no passwords). SSH is not running on a standard port and so is the Webmin/Virtualmin panel. I was the only person to access the server and having the necessary credentials.
What would be the right course of action now?
I will check my computers for possible malware/trojans, of course. But what "forensic" possibilities do I have to find out what might have happened? I never came across a situation like this and am just an ambitious amateur. I will re-install the server from scratch after some investigation, but I would really like to find out, how that was possible.
I have backups of all data, so no problem there and the server is powered off for the moment.
Thank you in advance for any helpful hints!
Comments
Do you have an IDS by any chance?
Logs will tell you a lot. Check for unknown processes running and recent file changes/new files (
findwith time parameters)If you wanted to do full forensics, you'd want to take a memory dump and hard drive image first - but that is probably unnecessary. And seeing as you already powered down the server, taking a memory dump after a fresh boot would not help anyway.
You want to find out answers to these:
You told us what measures you took to prevent hacking but didn't tell us which you didn't. It's not what you know is dangerous but what you don't know.
I assume webmin was used to host something to the outside world. Maybe your site had an exploit.
No, not really. There is CSF/LFD running on that server and the built-in brute force protection that comes with Webmin/Virtualmin and it is configured to be quite strict. But I suspect that both does not count as Intrusion Detection System.
Thank you very much for the detailed list of things to check. I will spend my weekend now going through your valuable advices!
I agree. I am still a bit shocked and surprised about that incident and will go through the four websites that are hosted on the server. There is no CMS installed, everything facing the outside world is plain HTML. But this is only what I think, I will have to verify that during the next hours. As you said, "It's not what you know is dangerous but what you don't know."...
By any chance the deleted stuff is in fact all the files under a virtual account?
If yes, do you use virtualmin built-in backup?
If yes, did this happen after a failed backup ?
I've faced this problem and it always happens after a failed backup, but honestly I never pursuit it much.
Indeed, it seems as if all missing data is just under one single virtual account. But not all data is missing. Only 2 complete directories, the rest seems intact.
Yes, I do.
No. At least not as far as I can see. I get an eMail every time that the daily backups runs and it never reported a problem. That does not mean that there wasn't any, but the report was always positive.
That seems like quite a bug then. I was thinking about switching to cPanel anyway, but I did not see the need to pay so much every month for a personal server without selling services. Webmin/Virtualmin did a good job for me during the last years, but if there is a quirk, then it's extraordinary annoying...
Again, thank you all! This has all been very helpful!
Did you read the whole email? some virtual hosts may have no errors and some may have an error. The same email will have them all.
Well, I guess it depends on how important are those projects for you, in my case, the nightly backups are enough and I can handle a couple hours downtime.
So I never looked into it (I should have, I know) and at the same time I don't think it's worth paying for cPanel. As for free panels, webmin & virtualmin still feel as the easiest, feature rich , free panel around.
As a sidenote, I've been using Webmin for quite a long time, and this issue only happened a few times.
I thought it was standard procedure to always wipe the o/s if you suspect a break in and copy files over that does not execute ... what changed?
Ideally you want to investigate how the attacker got in in the first place. Reinstall would be just a temporary solution if you make the same config mistake/install the same vulnerable software/use same compromised password or whatever - the attacker would just get in again.
Like FHR said...
However, I did as much research as I could during the weekend.
It may really have been the Virtualmin backup bug that @404error mentioned. The backup logs of the last 14 days do not report any malfunction, but who knows. Strange thing.
I will give up. Whatever happened - I won't find out. I have now setup the server from scratch, all credentials changed and I copied the /home directory over from the backup. I will keep an eye on it, but I guess the haunting is over.
However - Does anybody have a good (and possibly free) IDS to recommend that works well on Debian? Maybe I should install one to make the monitoring easier...
Again, thank you for all your help, this was very much appreciated!
Security Onion is a good place to start - https://securityonion.net/ It's a suite of multiple solutions.
I normally recommend Suricata - https://suricata-ids.org/ if you want to just use a point solution.
Bear in mind that an IDS against a persistent adversary is not a silver bullet.
I just finished a tutorial on how to "misuse" HAProxy to detect TCP port scans. I happen to use Debian, too. It probably won't take you more than 10 minutes to set it up.
http://botnet-tracker.blogspot.com/2018/10/tcp-port-scan-detection-with-haproxy.html
@birchbeer
I think that might be a little too much for the op. Even tho you have posted a free solution (the second one). Is probably that backup bug that was mentioned around.
@3606202
Did you had backups? Was it sensitive data? Is probably easier to wipe the server and start again, if possible don't use any panels, unless is something like Centminmod which is all CLI.
A good rule: If using any web control panel go with cPanel or Plesk.
Maybe... The OP's setup and technical proficiency wasn't clear to me. I was just offering 2 free suggestions which are commonly used since OP asked for free IDS solutions and indicated that the server has been rebuilt from scratch with new credentials. An IDS may aid in forensics if something like this occurs in future or possibly even detect an attack.
I agree it could be a defect in virtualmin but it's unusual for files to be unlinked during a backup.