Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Trying to get fail2ban to lookup IPs (blocklist or similar)
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Trying to get fail2ban to lookup IPs (blocklist or similar)

I'm dealing with a possible unique challenge. I like using fail2ban, it is lightweight and simple, but bots have become more sophisticated.

Currently, I'm facing thousands of IPs attempting to access URLs with malicious encoded input, which leads to overloading of webserver. I don't want to block these IPs based on patterns because they're too clever at evading such measures. Rate limiting isn't effective since they aren't hammering with the same IP. Instead, they're using multiple IPs from all over the world, making up to 4 requests per second at most, without exceeding this rate in, say, 5 minutes. This makes it challenging to automatically detect and block them vs legit users.

What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked. I don't want a massive 50 million IP blocklist in my ipset list, as that would create other problems. I want to block only those actually hitting my servers. After identifying them through an IP lookup, they should be added to the blacklist if certain criteria's are met. This means the first hit will get through, which is acceptable for me, but new ongoing hits would be blocked from that same IP.

What is your suggestion to my idea? and any possible solution how to achieve it ? :)

Comments

  • I'd just turn on Cloudflare and use WAF :smile:

  • @dedipromo said:
    I'd just turn on Cloudflare and use WAF :smile:

    CF is not an option here, and unfortunately CF doesn't block it. Too much go trough. Client wont upgrade to PRO as well.

    So i need to find a long term solution on server level.

  • UrDNUrDN Member

    If your system is being overloaded by random scanning scripts, then your system must be very bad. Fix it instead of adding layers of crap.

  • You can utilize CSF (ConfigServer Security & Firewall) and LFD (Login Failure Daemon) with virtually any number of IP addresses, even upwards of hundreds of thousands of /32 IPs, by using ipsets. CSF will automatically fetch and update these lists on a scheduled basis, removing the need for any manual intervention. Additionally, you have the flexibility to set per-connection behaviors for these lists, such as limiting the number of TCP connections per second.

    AbuseIPDB runs an excellent list platform, and contributing back to it is highly recommended for the benefit of the internet. However, it's important to note that, like all lists, it can flag CG-NAT gateways and widely shared IPs, potentially causing false positives. Consequently, if you choose to completely block requests from all listed IPs rather than rate-limiting and/or challenging them, you will end up blocking some legitimate users as well.

    Often, a good solution is to use Nginx (or a fork like Angie or OpenResty) and use the IP lists with a LUA module for browser challenge. There are numerous modules available, and if one becomes too simple for your particular bots, you can easily switch to another.

    Thanked by 3mrTom lowprofile 0xC7
  • @lowprofile said:
    What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked.

    Well since you already knew crowdsec, why not just use their bouncer integrations? e.g.: https://docs.crowdsec.net/u/bouncers/nginx

  • wuckwuck Member

    Read the logs and add pattern to f2b, you can't really avoid those anyway unless you get the Cloudflare option.

    You could setup https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker it's not hard to install and does a decent job

    Thanked by 2mrTom lowprofile
  • @UrDN said:
    If your system is being overloaded by random scanning scripts, then your system must be very bad. Fix it instead of adding layers of crap.

    For god sake. Please find another thread. You clearly don't know what this issue is about.

  • @wuck said:

    Read the logs and add pattern to f2b, you can't really avoid those anyway unless you get the Cloudflare option.

    You could setup https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker it's not hard to install and does a decent job

    Thanks i will look into it.

    This is looking very good - Thanks i will try it out

    @vicaya said:

    @lowprofile said:
    What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked.

    Well since you already knew crowdsec, why not just use their bouncer integrations? e.g.: https://docs.crowdsec.net/u/bouncers/nginx

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    Thanked by 1Andreix
  • @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    Thanked by 1techdragon
  • @concept said:

    @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    Alright, do you know if there is any performance penalty? Or can it do a post-lookup?

  • GSBRTGSBRT Member, Patron Provider

    Have you tried using a ruleset like OWASP?

  • @concept said:

    @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    @lowprofile said:

    @concept said:

    @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    Alright, do you know if there is any performance penalty? Or can it do a post-lookup?

    I use CrowdSec.

    It adds about 4% CPU load and 3% RAM on a 2VCPU/2GB storage server. It's not noticeable on any other VPS or server.

    It's easy to set up.

    However, you may want to edit your configurations to be a little more strict. I think the default for SSH brute force is ten attempts in ten seconds. This can be adjusted to ten attempts in a minute as an example.

    Thanked by 1lowprofile
  • wadhahwadhah Member
    edited January 28

    @lowprofile said:

    @concept said:

    @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    Alright, do you know if there is any performance penalty? Or can it do a post-lookup?

    From my limited experience crowdsec is better performance wise than fail2ban at the same jobish. But I have not had attacks as bad as yours so I have no idea how it would do in your system.

    EDIT: Is the project you ary trying to protect related to crypto in anyway?

  • @wadhah said:

    @lowprofile said:

    @concept said:

    @lowprofile said:

    I will look into this, i have not any experience with crowdsec and dont even know what a bouncer is on crowdsec context. I will read up

    "What I need is for fail2ban to scan the access log and perform an IP lookup against CrowdSec or another updated blocklist. If an IP is identified as abusive and ex. not from a specific country, it should be blocked."

    The bouncer does exactly what you are looking for.

    https://docs.crowdsec.net/u/bouncers/intro

    Alright, do you know if there is any performance penalty? Or can it do a post-lookup?

    From my limited experience crowdsec is better performance wise than fail2ban at the same jobish. But I have not had attacks as bad as yours so I have no idea how it would do in your system.

    EDIT: Is the project you ary trying to protect related to crypto in anyway?

    Not at all - Just a regular ecommerce site selling merchandises.
    I created a fail2ban regex and now it is banning them permanently. Thousands of ip.
    Load decreased by 10x

Sign In or Register to comment.