WTF?! How was I pwned?
A monitoring service showed that my KVM VPS was rebooted, and when I logged in /var/log was empty and there were several strange processes running: /etc/my.conf, /var/ssh.conf, /usr/bin/.sshd, /usr/bin/pythno, /usr/bin/dpkgd/ps ax. (I did save a few of these binaries before I nuked the box, if there's a way to safely examine them.)
I happened to have a screen with /var/log/auth.log tailed. The last entry is ominous:
Jun 21 03:59:48 XXXX login: pam_unix(login:auth): check pass; user unknown Jun 21 03:59:48 XXXX login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost= Jun 21 03:59:51 XXXX login: FAILED LOGIN (1) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Wait, /dev/tty1? Isn't that the console connection? How would anyone connect there?
The down alert came in seconds later at 04:00:37 UTC. It went back up at 04:04:38 UTC.
More importantly, what did I do wrong to allow this to happen? I've been admin'ing VPSs for years, and I've never had something like this happen.
Background: Installed debian buster a few days ago (used the provider's Plesk VNC console to select the netinst kernel and initrd through grub), no root account, no services other than ssh at install. On first boot the usual: configure sshd to a non-standard port, disable sshd password auth, sshd AllowUsers to my account only, installed ntp, nginx, certbot, and munin. Configured iptables to allow tcp/80, tcp/443, udp/123, and tcp/ssh port through. Pretty sure I even disabled the VNC console through the Plesk control panel when I was done. It's an idler--there really wasn't anything running on it.
My account with the host (had!) a unique 64 char password and 2FA (TOTP) is active.
I'm stuck on that /dev/tty1 entry...all the usual login attempts come in through an IP to sshd, but not that one. What am I missing here?
Thanks in advance!