All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Please help - Hadn't connected to my VPS servers in weeks, now find most are compromised
I have several different servers (some are CentOS, but most are Debian) with multiple different providers. Over the past weeks-month, I became quite busy with life and didn't have time to do anything with them; my VPS are almost always idle. I did try to connect to some a couple weeks ago and couldn't, but didn't think much of it and didn't have the time to look into it.
Now, I'm finding that many of my servers are offline (I haven't had time to go through all of them fully, but so far, at least one was suspended for outgoing DOS and two are apparently still online, though they have used several-fold the monthly bandwidth allotment). When I tried to power on one of the VPS, it always becomes unresponsive within minutes and I can see that there are some processes that are using up the CPU and increasing the load.
One such process is as follows (if left unchecked, its CPU usage jumps to hundreds in minutes, but manually killing the processes like that just spawn new ones with the same behaviour). This is an example from one VPS (I have powered it down now) after I already changed the root password, removed my authorized_keys file, and rebooted the server several times:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3864 42.2 0.2 59552 1896 ? Ssl Jan11 16685:09 cd /etc
I have a couple questions:
1) Is there any easy way to see what IPs have been connecting to/logging into my various servers and block them? I will obviously have to do a complete rebuild of my servers.
2) Is there any way to boot up the VPS (CentOS or Debian) in something like Windows' "safe mode with networking" so that it boots up with just the bare minimum required to run and accept SSH connections? I cannot recall for sure, but I am guessing that there will be at least one or two servers with a couple files I would want to save (if possible).
Thanks!
ETA: To beat you all to the punch, yes, I know I'm an idiot.
Comments
Personally, I'd ask for a backup of the affected VPS's, rebuild new ones, secure them better, and dig through the backup for the files that you are missing. You will also have access to all the log files from the VPS for your forensic research. Some providers offer the ability to backup and download said backup through their panel, if not, you'll have to raise a ticket with each one.
This.
Steps
1. Make backups of your important data
2. Delete all affected VMS
3. Rebuild VMS
4. Properly secure VMs this time. Install things such as fail2ban, and make strong passwords.
5. Rebuild your applications
6. Monitor logins.
Ticketing with your provider kindly ask them to pack your files. Then rebuild the VM.
I am curious to know, who your providers are?
Better secure the vps a bit more next time!
http://lowendtalk.com/discussion/20572/guide-basic-steps-to-secure-your-ubuntu-debian-server/p1
This is usually good enough, i have yet to witness any security breaches.
Thanks for the feedback, everyone! If a provider does not want to create a backup for me (I have not yet asked, but sometimes it takes a while to hear back from a provider), is there any way to boot up a VPS in something like Windows' "safe mode with networking" so that it boots up with just the bare minimum startup items required to run and accept SSH connections? Currently, I can't even backup a file that is a couple MB because the CPU usage spikes to ridiculous levels so quickly.
Once I rebuild the affected servers (and probably also the few that seem to not be compromised), I plan to do the following:
apt-get install sudo apt-get update && apt-get upgrade (auto-check and install new updates) apt-get install apticron unattended-upgrades Install something to block repeated login attempts (either denyhosts - only does ssh - or fail2ban, which can do ssh and other processes) sudo apt-get install fail2ban Can override the default configuration by creating a new jail.local file. sudo nano /etc/fail2ban/jail.local http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Configuration Set the bantime variable to specify how long (in seconds) bans should last. Set the maxretry variable to specify the default number of tries a connection may be attempted before an attacker’s IP address is banned. ctrl+X, then Y sudo service fail2ban restart sudo apt-get install denyhosts config file is located at /etc/denyhosts.conf Create a sudo user, add user to admin group, add authorized_keys, & disable root user password login adduser newuser (passwd newuser) usermod -a -G sudo newuser sudo nano /etc/ssh/sshd_config Change the PermitRootLogin setting to no ctrl+X, then Y sudo service ssh restart Can change SSHd from port 22 to another free port <1024 nano /etc/ssh/sshd_config change Port 22 to Port xxx service ssh restartIf you wanted you could lock it to certain users also "allowusers" in sshd config.
Best thing you can do to protect SSH access is to disallow passwords and use key based authentication.
As for your problem of "Safe mode access" then it depends on the type of VPS. If it's openvz then the provider can copy any files for you. If it's KVM or Xen then mount a rescue image ISO and boot from that. Then you can mount and access the file system.
You should also use public key authentication rather than password authentication. With password authentication, your password is transmitted over the Internet. With a public/private key, the password for your private key never leaves your workstation. If your workstation runs Windows, see https://linuxinstructions.wordpress.com/
It is weird that a lot of your different servers with different providers were compromized within a month. I think this is something you should search for, because it cannot be such a coinsidence. Maybe something you upload or, I don't know, your pc, causing the holes? Do you upload a certain software to your vps's? Do you use a default password like 123456?
On (almost) idling boxes i'd recommend debian with a restrictive firewall config & unattended-upgrades - that way your machine will roll security updates even if you don't log in for a while...
(edit: or better yet, if it's fully idle shut down the VPS on the panel!)
In your cases fresh installs + better config & updates this time and you're set
Fail2Ban and changing SSH port, keeping upgraded etc. All part of being a hardened VPS.
http://www.rackaid.com/blog/how-to-harden-or-secure-ssh-for-improved-security/
http://codex.wordpress.org/Hardening_WordPress
http://wiki.centos.org/HowTos/OS_Protection
https://wiki.debian.org/Hardening
iptables support is limited on some Ovz vps.
A good solution I found is to use /etc/hosts.allow and /etc/hosts.deny
Public keys are great but I like to keep password-based accesses too. Just in case.
Outgoing DoS, high traffic and the resource hungry processes may be because of an insecure ssh password, an exploitable, outdated service or an insecure password of another service.. So they has just been hacked and someone made them doing something.
Thanks for the feedback, everyone! I have ticketed all the hosts of the servers that appear to be affected and also changed the passwords on the servers that don't seem to be affected. I think that the passwords might have been compromised (while they weren't top-10 or dictionary passwords, let's just say...my decision surrounding password choice was not optimal). I did also have auth keys enabled (the same one on all the servers), but did not disable root password login in case I need to access a server from random places (i.e. when on vacation or at a conference). Currently, I have somewhere around 10 VPS spread out over 7 different hosts.
I'm working on getting a backup/dump of each VPS from the hosts and then having them rebuild the server with a fresh OS installation.
ETA: All VPS are OpenVZ; I used to have a KVM and some dedi slices, but those hosts deadpooled/disappeared.
@user123
As the others said, disable password auth and use keys only.