New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Is this an SSH brute force attack? Can you brute force a key auth?
When I woke up today morning, I noticed that my average CPU consumption has jumped from around 10% to 50%, and most of the extra load was generated by fail2ban.
I turned off port 22 ipv4 and cpu consumption went back to normal.
Checked auth.log and it's full of such entries.
What exactly is this guy trying to do? Can you really brute force key-based SSH authentication?
Mar 6 00:37:24 localhost sshd[223106]: Connection closed by [my provider IP] port 58594 [preauth]
Mar 6 00:37:24 localhost sshd[223108]: Connection closed by [my provider IP] port 58596 [preauth]
Mar 6 00:37:24 localhost sshd[223109]: Connection closed by 128.199.71.187 port 60084 [preauth]
Mar 6 00:37:24 localhost sshd[223110]: Connection closed by 128.199.71.187 port 60086 [preauth]
Mar 6 00:37:24 localhost sshd[223114]: Connection closed by [my provider IP] port 58598 [preauth]
Mar 6 00:37:24 localhost sshd[223117]: Connection closed by [my provider IP] port 58600 [preauth]
Mar 6 00:37:24 localhost sshd[223116]: Connection closed by 128.199.71.187 port 60088 [preauth]
Mar 6 00:37:24 localhost sshd[223118]: Connection closed by 128.199.71.187 port 60090 [preauth]
Mar 6 00:37:24 localhost sshd[223122]: Connection closed by [my provider IP] port 58602 [preauth]
Mar 6 00:37:24 localhost sshd[223124]: Connection closed by [my provider IP] port 58604 [preauth]
Mar 6 00:37:24 localhost sshd[223125]: Connection closed by 128.199.71.187 port 60094 [preauth]
Mar 6 00:37:24 localhost sshd[223127]: Connection closed by 128.199.71.187 port 60096 [preauth]
Mar 6 00:37:24 localhost sshd[223130]: Connection closed by [my provider IP] port 58606 [preauth]
Mar 6 00:37:24 localhost sshd[223132]: Connection closed by [my provider IP] port 58608 [preauth]
Mar 6 00:37:24 localhost sshd[223136]: Connection closed by [my provider IP] port 58610 [preauth]
Mar 6 00:37:24 localhost sshd[223134]: Connection closed by 128.199.71.187 port 60098 [preauth]
Mar 6 00:37:24 localhost sshd[223135]: Connection closed by 128.199.71.187 port 60100 [preauth]
Mar 6 00:37:24 localhost sshd[223140]: Connection closed by [my provider IP] port 58612 [preauth]
Mar 6 00:37:24 localhost sshd[223142]: Connection closed by [my provider IP] port 58614 [preauth]
Mar 6 00:37:24 localhost sshd[223144]: Connection closed by 128.199.71.187 port 60104 [preauth]
Mar 6 00:37:24 localhost sshd[223145]: Connection closed by 128.199.71.187 port 60106 [preauth]
Mar 6 00:37:24 localhost sshd[223147]: Connection closed by [my provider IP] port 58616 [preauth]
Mar 6 00:37:24 localhost sshd[223150]: Connection closed by [my provider IP] port 58618 [preauth]
Mar 6 00:37:24 localhost sshd[223152]: Connection closed by 128.199.71.187 port 60112 [preauth]
Mar 6 00:37:24 localhost sshd[223154]: Connection closed by [my provider IP] port 58620 [preauth]
Mar 6 00:37:24 localhost sshd[223153]: Connection closed by 128.199.71.187 port 60114 [preauth]
Mar 6 00:37:24 localhost sshd[223158]: Connection closed by [my provider IP] port 58622 [preauth]
Mar 6 00:37:24 localhost sshd[223159]: Connection closed by 128.199.71.187 port 60118 [preauth]
Mar 6 00:37:24 localhost sshd[223163]: Connection closed by [my provider IP] port 58624 [preauth]
Mar 6 00:37:24 localhost sshd[223161]: Connection closed by 128.199.71.187 port 60120 [preauth]
Mar 6 00:37:24 localhost sshd[223166]: Connection closed by [my provider IP] port 58626 [preauth]
Mar 6 00:37:24 localhost sshd[223168]: Connection closed by 128.199.71.187 port 60122 [preauth]
Mar 6 00:37:24 localhost sshd[223169]: Connection closed by [my provider IP] port 58628 [preauth]
Mar 6 00:37:24 localhost sshd[223171]: Connection closed by 128.199.71.187 port 60124 [preauth]
Mar 6 00:37:24 localhost sshd[223174]: Connection closed by [my provider IP] port 58630 [preauth]
Mar 6 00:37:24 localhost sshd[223176]: Connection closed by [my provider IP] port 58632 [preauth]
Mar 6 00:37:24 localhost sshd[223177]: Connection closed by 128.199.71.187 port 60128 [preauth]
Mar 6 00:37:24 localhost sshd[223181]: Connection closed by [my provider IP] port 58634 [preauth]
Mar 6 00:37:24 localhost sshd[223179]: Connection closed by 128.199.71.187 port 60130 [preauth]
Mar 6 00:37:24 localhost sshd[223184]: Connection closed by [my provider IP] port 58636 [preauth]
Mar 6 00:37:24 localhost sshd[223185]: Connection closed by 128.199.71.187 port 60132 [preauth]
Mar 6 00:37:24 localhost sshd[223188]: Connection closed by [my provider IP] port 58638 [preauth]
Mar 6 00:37:24 localhost sshd[223191]: Connection closed by [my provider IP] port 58640 [preauth]
Mar 6 00:37:24 localhost sshd[223190]: Connection closed by 128.199.71.187 port 60134 [preauth]
Mar 6 00:37:24 localhost sshd[223194]: Connection closed by [my provider IP] port 58642 [preauth]
Mar 6 00:37:24 localhost sshd[223196]: Connection closed by [my provider IP] port 58644 [preauth]
Mar 6 00:37:24 localhost sshd[223197]: Connection closed by 128.199.71.187 port 60140 [preauth]
Mar 6 00:37:24 localhost sshd[223199]: Connection closed by 128.199.71.187 port 60144 [preauth]
Mar 6 00:37:24 localhost sshd[223201]: Connection closed by [my provider IP] port 58646 [preauth]
Mar 6 00:37:24 localhost sshd[223204]: Connection closed by [my provider IP] port 58648 [preauth]
Mar 6 00:37:24 localhost sshd[223206]: Connection closed by [my provider IP] port 58650 [preauth]
Mar 6 00:37:24 localhost sshd[223207]: Connection closed by 128.199.71.187 port 60146 [preauth]
Mar 6 00:37:24 localhost sshd[223209]: Connection closed by 128.199.71.187 port 60148 [preauth]
Mar 6 00:37:24 localhost sshd[223211]: Connection closed by [my provider IP] port 58652 [preauth]
Mar 6 00:37:24 localhost sshd[223214]: Connection closed by [my provider IP] port 58654 [preauth]
Mar 6 00:37:24 localhost sshd[223216]: Connection closed by [my provider IP] port 58656 [preauth]
Mar 6 00:37:24 localhost sshd[223217]: Connection closed by 128.199.71.187 port 60152 [preauth]
Mar 6 00:37:24 localhost sshd[223218]: Connection closed by 128.199.71.187 port 60154 [preauth]
Mar 6 00:37:24 localhost sshd[223222]: Connection closed by [my provider IP] port 58658 [preauth]
Mar 6 00:37:24 localhost sshd[223224]: Connection closed by [my provider IP] port 58660 [preauth]
Comments
That's just another normal day on the wild jungle web!
"Can you brute force a key auth?"
You can if its SHA1 and you have a lot of dedication or get lucky. I imagine they're hoping that's the case.
Note that all the offered key algorithms involve SHA1 in some way. They are attempting insecure algorithms exclusively.
sorry, the log bits I pasted originally was from Mar 1. I've updated the log bits from today. There are dozens of such requests from a single IP per second. After a few seconds, the IP changes. Mostly Chinese and Russian IP addresses.
I'm used to this, of course. But I've never seen anyone try to brute force a key-based authentication. Normally, one sees people checking to see if password authentication is enabled or not, and when they see it's not, I guess they just move on. This is new for me. Or probably I never noticed. Even today, I noticed only because I saw CPU levels remain elevated for 3-4 hours.
IIRC, years back there was a bug in ssh-keygen on debian that reduced the keyspace dramatically.
Actually the bug was in openssl called by ssh-keygen:
https://www.debian.org/security/2008/dsa-1576
So they might be trying to exploit something like that too.
So many requests per second
It's enough to affect my server's performance. Anyway, now that i've closed the port, it's all good.
Set wireguard and mount sshd on internal IP. Than you will avoid such problems entirely.
That is pretty normal. If you set ssh to a different port though it will reduce the number of attempts dramatically.
Those log messages don't mean the clients are attempting SSH public key authentication. Those are probably just the typical password authentication brute force attacks.
If your server have IPv6 then generate an IPv6 address and listen on it only, no more ssh attacks.
I doubt these are brute force attacks on SSH keys. It's rather likely that you have password authentication still enabled despite using SSH keys.
Try changing "PasswordAuthentication" to no in
/etc/ssh/sshd_configand see if the attacks persist. In my observations, SSH bots disconnect immediately as soon as they see key based auths as the only one being offered.Changing it to No and restarting sshd is the first thing I do when I get a VPS. I did double check it, and it was not enabled.
What's interesting is that today evening, another server of mine which is hosted in the same datacenter and holds our database started getting attacked this way. It struck me as quite surprising since that server's IP is not made public in any way (as in, it's accessible from outside, but no domain is configured to it.)
It's likely therefore that the hacker is trying all IPv4 addresses in a series.
This is what I'm doing right now. However, the funny thing is, there's a server on which the IPv6 address is also configured with a domain. On that server alone, even the IPv6 port was getting attacked and I've decided to use only the provider's VNC on it.
I guess wireguard is something I should look at eventually. But I do wonder if the wireguard port too won't be attacked somehow.
It's also been my experience. Is there any other authentication method that may be on? As in, anything besides these two?
you interpret it wrong.
from the logfiles you can see, that no authentication try is happening at all, instead the connection is dropped on pre-auth. this exactly means, the bot-net is requesting regular password auth and because that's not available sshd rejects already before any username and such is send.
the bot-net obviously does not analyze nor care for the reason it did not get through. instead continues to loop through its list throwing one try after another...
you should check, if fail2ban actually does anything about it at all. because there is no 'authentication failed' in the logfile I'd assume it does not pick up all these attempts as malicious. simply check iptables if the offenders get banned at all.
easiest way to get rid of the noise (for a while) is changing the ssh port. (and making sure fail2ban really blocks on these logentries).
Yes. That's another curious thing I noticed. Only 80 IPs banned in the last 1 week. Fail2ban is not really banning anything. But when I do top, I see it up there as one of the biggest CPU hogs when this happens.
The only other explanation I can think of is that this is some kind of stealth DDOS attack. Instead of using HTTP, perhaps they're using SSH as a more discrete mode of attack.
you have many friends.
please read my post again. it is a very dumb bot throwing tons of requests at your ssh server. nothing stealth, nothing uncommon. it simply does not get to the point of sending any kind of credentials because sshd rejects beforehand...
imagine a normal failed login looking like this (with password enabled):
with password disabled but trying to:
the bot simply does not care that it gets rejected because password auth is not available and starts one try after the other:
with this you also can understand why fail2ban is not working correctly. it usually parses the 'auth failed!' in the log file - which does not happen here, because the connection is dropped before that even can happen.
that it still causes such high load is because the high amount of log entries and fail2ban being triggered via inodes for every change. with the auth.log quickly growing and fail2ban analyzing it on each new entry, of course the load will sky-rocket without anything happening, because the filter doesn't match and no ban is happening.
you need to change the ssh filter mode in fail2ban to 'aggressive' or modify the filter regex itself.
this is a good read: https://serverfault.com/questions/686422/modify-fail2ban-failregex-to-match-failed-public-key-authentications-via-ssh/686436
though the question also assumes wrongly failed key-auth attempts. only the first comment to the second answer notices that this is a wrong assumption.
The default fail2ban settings likely unban them after so many hours/days.
This it what I did in fail2ban etc/fail2ban/jail.local
[sshd]
port = ssh , new port # here
enabled = true
maxretry = 3
findtime = 1d
bantime = 4w
ignoreip = 127.0.0.1/8 my IP# 00.0.0.00
Every auth system can be brute force. How ever, to brute force a key, you usually need 2^n operations. n for the key size. Some algorithm has been prove not secure that can be collision attack. Still need 2^(n/2). Possible but not doable.