All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Trying to get fail2ban to lookup IPs (blocklist or similar)

I'm dealing with a possible unique challenge. I like using fail2ban, it is lightweight and simple, but bots have become more sophisticated.
Currently, I'm facing thousands of IPs attempting to access URLs with malicious encoded input, which leads to overloading of webserver. I don't want to block these IPs based on patterns because they're too clever at evading such measures. Rate limiting isn't effective since they aren't hammering with the same IP. Instead, they're using multiple IPs from all over the world, making up to 4 requests per second at most, without exceeding this rate in, say, 5 minutes. This makes it challenging to automatically detect and block them vs legit users.
What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked. I don't want a massive 50 million IP blocklist in my ipset list, as that would create other problems. I want to block only those actually hitting my servers. After identifying them through an IP lookup, they should be added to the blacklist if certain criteria's are met. This means the first hit will get through, which is acceptable for me, but new ongoing hits would be blocked from that same IP.
What is your suggestion to my idea? and any possible solution how to achieve it ?
Comments
I'd just turn on Cloudflare and use WAF
CF is not an option here, and unfortunately CF doesn't block it. Too much go trough. Client wont upgrade to PRO as well.
So i need to find a long term solution on server level.
If your system is being overloaded by random scanning scripts, then your system must be very bad. Fix it instead of adding layers of crap.
You can utilize CSF (ConfigServer Security & Firewall) and LFD (Login Failure Daemon) with virtually any number of IP addresses, even upwards of hundreds of thousands of /32 IPs, by using ipsets. CSF will automatically fetch and update these lists on a scheduled basis, removing the need for any manual intervention. Additionally, you have the flexibility to set per-connection behaviors for these lists, such as limiting the number of TCP connections per second.
AbuseIPDB runs an excellent list platform, and contributing back to it is highly recommended for the benefit of the internet. However, it's important to note that, like all lists, it can flag CG-NAT gateways and widely shared IPs, potentially causing false positives. Consequently, if you choose to completely block requests from all listed IPs rather than rate-limiting and/or challenging them, you will end up blocking some legitimate users as well.
Often, a good solution is to use Nginx (or a fork like Angie or OpenResty) and use the IP lists with a LUA module for browser challenge. There are numerous modules available, and if one becomes too simple for your particular bots, you can easily switch to another.
Well since you already knew crowdsec, why not just use their bouncer integrations? e.g.: https://docs.crowdsec.net/u/bouncers/nginx
Read the logs and add pattern to f2b, you can't really avoid those anyway unless you get the Cloudflare option.
You could setup https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker it's not hard to install and does a decent job
Maybe this article would help: How to Mitigate Sophisticated Bot Attacks Using Fail2ban and CrowdSec on Debian/RHEL
For god sake. Please find another thread. You clearly don't know what this issue is about.
@wuck said:
Thanks i will look into it.
This is looking very good - Thanks i will try it out
I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up
"What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."
The bouncer does exactly what you are looking for.
https://docs.crowdsec.net/u/bouncers/intro
Alright, do you know if there is any performance penalty? Or can it do a post-lookup?
Have you tried using a ruleset like OWASP?
I use CrowdSec.
It adds about 4% CPU load and 3% RAM on a 2VCPU/2GB storage server. It's not noticeable on any other VPS or server.
It's easy to set up.
However, you may want to edit your configurations to be a little more strict. I think the default for SSH brute force is ten attempts in ten seconds. This can be adjusted to ten attempts in a minute as an example.
From my limited experience crowdsec is better performance wise than fail2ban at the same jobish. But I have not had attacks as bad as yours so I have no idea how it would do in your system.
EDIT: Is the project you ary trying to protect related to crypto in anyway?
Not at all - Just a regular ecommerce site selling merchandises.
I created a fail2ban regex and now it is banning them permanently. Thousands of ip.
Load decreased by 10x