New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Edit: Reading the ticket they are confident there was no compromise on their end. And to clarify I'm not blaming them and at least they're investigating.
maybe they were just looking at a basic top output and saw someone trying to brute force your VPS instead. as you said you did nothing to lock it down to it was ripe and open to that.
You know the old
There were 1182729 failed login attempts since the last successful loginI did have sshguard installed at least. I will say they were very responsive and informative and did a good job in handling my concens. It definitely sounded like there was a vulnerabilty on the VPS and not something larger as I initially speculated.
Did you disable password auth and switch to using SSH keys, or did you keep password auth enabled? If you kept it enabled, was the password sufficiently complex? Did you check the auth logs to ensure it wasn't someone that brute forced the password?
Everything was default per their Debian image. In addition to running SSH guard, the password doesn't appear to be one that anyone could really brute force easily (a long seemingly random generated alphanumeric string created by their system). I also didn't install any externally accessible service from memory. I couldn't check any logs since the service was suspended. In any case they were prompt with their response and I feel like they did a really good job handling the situation, initially I was just extremely frustrated because it was a jarring experience having your server compromised (it feels like a violation). I consider the situation resolved.
Was it a default password sent via email? Email is not really a secure transport mechanism (particularly as TLS is not enforced by any servers) so in general you should change any passwords received via email
Yes.
This was honestly one of the first thoughts that went through my head after assuming it was a system error. And I know better than to not change the default password. It's definitely still a major possibility I think.
would restrict access to using key authentication prevent this?
I wish providers didn't use default passwords. DigitalOcean does it well... They let you provide SSH public keys, in which case they don't use a default password at all. I still personally prefer installing directly from ISO though.
Installers for the newer versions of Ubuntu can automatically import your SSH public key from your GitHub account, which is nice.
Yeah, you should do that. That's the first thing I do with any server. Make sure your SSH key works fine (so you don't lock yourself out), then set
PasswordAuthentication noin/etc/ssh/ sshd_configThe worst thing is many providers email customer their server password in plain text. I hate it every time!
I had a system in LA that was suspended a few weeks back. The reason was something like 150 open SMTP connections. I told them that I had never logged into the box and that sendmail must have been running by default. They claimed they disabled it in their image. After disputing the suspension, a tech logged in and said, "huh, sendmail is running. must have been something that stopped prematurely in the imaging." I didn't buy it, since I have two other Virmach VPS that are unused and running sendmail by default.
To their credit, they did side with me and credited me a full month of service.
Unfortunately, I got another suspension notice today for the same machine where they said it was a DDoS attack. Again, never logged in. Not running any services. Never did anything with the server. May be bad luck. Not sure.
Interesting that both times were with the same VPS in LA. I have 25+ services with Virmach. Never had a single suspension.
the end is wooohooo
I have a VPS with them running Windows Server 2016. This one has two cores (BF 2018 deal) and when I got the machine, I ran Windows Update thinking it will finish in a few minutes or at most may be 15, 20 minutes. But it took more than 2 fucking hours and whole time I was dreading that I will receive suspension notice as CPU was around 50% to 90%. But nope......no suspension, nothing. Point is they are not unreasonable and not actively trying to suspend your VPS on a drop of hat.
We are not owned by ColoCrossing.
The fees have always been there and they are only charged (and not waived) in clear cases of abuse. This means a suspension or blacklistings that required manual intervention or caused much harm -- not for unpaid invoices. In terms of payment, we do charge the fee in case of chargebacks. And finally there's a fee in any case (catch all) where we do something we normally would not even offer. We just have that in there to let you know that we reserve the right to bill you for staff time if we make big exceptions to our policies that cost a lot of time -- this is almost always optional. All these fee amounts are reasonable, even if they seem high due to the low price of our service. If a human is involved the cost will be much higher than the cost of a small automatically provisioned virtual server.
Customers are also always responsible for what occurs on their service. However, if you are compromised in one way or another we usually help you out and waive the fee as long as you cooperate with us and make some reasonable changes (password change, re-install.) This is at our discretion so unfortunately there will be some cases where a support agent may feel as if the situation is a little more serious.
OP was most likely also suspended for abuse and/or a payment dispute but without any further information I can't confirm.
I'm happy that VirMach publishes rather precise parameters for what is acceptable. When trying to be a "good citizen" it is helpful to know what that actually means. I have a script that polls /proc/loadavg and /proc/stat and throttles things if I'm getting anywhere near their limits. Unfortunately iops can't be properly monitored on OVZ so I've got to wing that.
When they say too much about Kolo (for football fans)
Well, I am here and no can't be here all the time.
Well maybe the name should be something different, "suspension/admin fee" is confusing as hell but thank you.
It could be there is a vulnerable service in one or more of their default OS templates, and/or the service is using default credentials. For example, the log provided to me showed Apache running and I have never installed nor used Apache before (if I were to use a HTTPD I'd use nginx). I cancelled my service / requested my account be deleted immediately after seeing the other report of an idle VPS suspended and have moved on. Hopefully they are able to find out how these servers are being accessed and/or compromised.
I can confirm httpd is running by default, at least in CentOS 6.x. Same for sendmail.
"shitweasel fee" (tm)
Umm, turn it the fuck off? An unpatched server is going to get hacked if you leave it sitting there unmanaged, not monitored. Your reaction and surprise tells me you shouldn't be in control of servers.
Lol, that might seems a bit harsh but ...
that is usually the first thing I do with a new vps: turn it off until I can set it up properly.
Then - when I am ready to give the little critter some quality time - if I am new to that particular provider I might boot it once from the host's template to test that it works and to check the "default" network config.
then I re-install from an ISO (assuming it's KVM). Or - if OVZ - at least set a strong root password. Then:
Takes all of maybe 20 minutes to install from ISO
And maybe 2 minutes to finagle the sshd_config as per above
If the vps is going to do anything more than idling, I'll setup a firewall to limit incoming connections to only a few trusted IP address while I configure whatever more complicated network-facing bits.
It's not rocket surgery, just the bare minimum to give my vps a decent chance of not becoming low-hanging fruit for the brute-force probulation that is well nigh inevitable within a few minutes of connecting to the internet.
Sounds like we can make a LET community script called as
secure.shfor beginners. Once the script is executed, it will perform most best practices and they will receive all the informations on how to login in the next reboot.While basically a good idea it might be better to leave them the option to learn themselves.
Maybe a tutorial or sth would be better.
It's a tempting notion, but my inner @bsdguy is screaming ... "security is a process!"
(also "let me out!" and "@WSS did nothing wrong!" and so on, and so forth ...)
so, (now channeling the ghost of @WSS) maybe better name any attempt at such a script changemydiaper.sh ...
Details for different distros might end up just similar enough to be especially confusing.
I guess bottom line for providers may be: guide your users' secure onboarding or be prepared for various degrees of disaster. Write good docs, or be prepared to see those (maybe more valuable) users who actually care to learn a bit flocking to some other host's documentation ecosystem (and maybe end up going with those other hosts instead) ....
https://www.digitalocean.com/community/tutorials/7-security-measures-to-protect-your-servers
https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-16-04
https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-debian-9
https://blog.digitalocean.com/cloud-firewalls-secure-droplets-by-default/
EDIT2:
Also keep in mind the adage "no good deed shall go unpunished" ...
If a simple securitah script were even possible to write, it would be a (low-end, high-stakes) bear to support.
Something as simple as
could end up leading to all kinds of fun after a node reboots and (possibly) starts up the vps again automatically.
(If I have LUKS full-disk encryption setup, it will ask for a passphrase before fully booting - but the point is I want the vps to stay shut-down until such time that I set that up. So a bit of a catch-22 there.)
EDIT3:
Anyhoo ... wishing it were easier.
Recognizing that even the stuff that's "easy once you know how" - is maybe 20 minutes of work - but (for me) only after well over 20 years of noodling around with the stuff. So it's best to be humble about it - the more I learn, the more I realize how little I actually know.
And the process that "works for me" is most likely not necessarily right for people doing a setup that involves (for example) supporting more than one user, and so forth. (This is my "idler" setup, after all.)
All that said - after
pooping all overgently critiquing @FAT32's suggestion, it does occur to me that there is more (what's left of) the community here can do to guide basic education.Maybe some sort of "leveling up" low-end certification process.
Perhaps following a checklist, but being able to analyze and adapt as appropriate.
I'm a bit short on time at this moment to write much more, but am hopeful there may be more good suggestions by the time I get back to the keyboard.
A script to at least identify common holes in security setup would be quite useful (and I imagine there are more than a few out there in the white-hat universe - something geared for beginners would indeed be the bees knees.)
Have you tried telling someone that gives a fuck?
I've merely been mucking around for a few years so obviously there's a lot I don't know. I always use default OS templates but there are some general rules I abide by to minimise the major risks of server compromise:
I choose only Debian/Ubuntu because for noobs like me, updating, configuring firewall rules (UFW) and protecting SSH from brute force attacks (Fail2ban) are much easier to implement.
If i were to cancel my cable service, i'd be charged a suspension fee.
Because they have to send someone to a pole in order to cut my analog service cable.
A fully automated hosting provider on the other hand does not have to do this.
One can argue that the automation is not free and that's absolutely true, but it's being paid from the successful orders themselves.