Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Massive Layer7 attack, more than 33 hours
New on LowEndTalk? Please Register and read our Community Rules.

Massive Layer7 attack, more than 33 hours

FlorinMarianFlorinMarian Member, Patron Provider

Good evening!

Since yesterday morning I have been facing a major problem related to the security of my website (whmcs), waking up suddenly with an attack coming from nowhere that does not stop.

From the information collected through CloudFlare / iptables / IDS, I have the following conclusions:
1. Attacks use tens of thousands of different IP addresses
2. The attacks come from almost every country in the world (I also saw IP addresses in my country, Romania)
3. The attack is so clever that it runs 24/7 (it resumes automatically when I turn off LiteSpeed ​​for a certain amount of time)
4. The attack uses enough intensity to keep my processor 100% busy (I tested i7-6700K, Ryzen 3800X, 2x E5-2699 V3), all were sooner or later exhausted but without using for example 8000 connections if the processor took only 800 to overload.
5. CloudFlare rules both WAF and "Rate limiting rules" although they block thousands of IP addresses per minute, still pass enough IP addresses through their tests and reach the final target
6. OVH Game or Advanced 10Gbps does not detect the attack at all
7. Voxlity Layer 7 AntiDDoS blocks many IP addresses but still gets a few hundred connected in parallel and paralyzes the entire web server

Last iptables rules applied:

#!/bin/bash

modprobe ip_conntrack

/sbin/sysctl -w net.netfilter.nf_conntrack_max=4194304
echo "524288" > /sys/module/nf_conntrack/parameters/hashsize


# ACCEPT LOOPBACK
iptables --append INPUT -i lo -j ACCEPT
# FIRST PACKET HAS TO BE TCP SYN
iptables --append INPUT -p tcp ! --syn -m state --state NEW -j DROP

# DROP FRAGMENTS
iptables --append INPUT -f -j DROP

# DROP XMAS PACKETS
iptables --append INPUT -p tcp --tcp-flags ALL ALL -j DROP

# DROP NULL PACKETS
iptables --append INPUT -p tcp --tcp-flags ALL NONE -j DROP

# DROP EXCESSIVE TCP RST PACKETS
iptables --append INPUT -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/second --limit-burst 2 -j ACCEPT

# DROP ALL INVALID PACKETS
iptables --append INPUT -m state --state INVALID -j DROP
iptables --append FORWARD -m state --state INVALID -j DROP
iptables --append OUTPUT -m state --state INVALID -j DROP

# DROP RFC1918 PACKETS
iptables --append INPUT -s 10.0.0.0/8 -j DROP
iptables --append INPUT -s 172.16.0.0/12 -j DROP
iptables --append INPUT -s 192.168.0.0/16 -j DROP

# DROP SPOOFED PACKETS
iptables --append INPUT -s 169.254.0.0/16 -j DROP
iptables --append INPUT -s 127.0.0.0/8 -j DROP
iptables --append INPUT -s 224.0.0.0/4 -j DROP
iptables --append INPUT -d 224.0.0.0/4 -j DROP
iptables --append INPUT -s 240.0.0.0/5 -j DROP
iptables --append INPUT -d 240.0.0.0/5 -j DROP
iptables --append INPUT -s 0.0.0.0/8 -j DROP
iptables --append INPUT -d 0.0.0.0/8 -j DROP
iptables --append INPUT -d 239.255.255.0/24 -j DROP
iptables --append INPUT -d 255.255.255.255 -j DROP

# ICMP SMURF ATTACKS + RATE LIMIT THE REST
#iptables --append INPUT -p icmp --icmp-type address-mask-request -j DROP
#iptables --append INPUT -p icmp --icmp-type timestamp-request -j DROP
#iptables --append INPUT -p icmp --icmp-type router-solicitation -j DROP
#iptables --append INPUT -p icmp -m limit --limit 2/second -j ACCEPT
iptables --append INPUT -j REJECT -p icmp --icmp-type echo-request

# ACCEPT SSH
iptables --append INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT

# DROP SYN-FLOOD PACKETS
iptables --new-chain RATE-LIMIT
iptables --append INPUT --protocol tcp --match conntrack --ctstate NEW --jump LOG --log-prefix "NEW TCP CONN: "
iptables --append INPUT --match conntrack --ctstate NEW --jump RATE-LIMIT
iptables --append RATE-LIMIT \
  --match hashlimit \
  --hashlimit-mode srcip \
  --hashlimit-upto 50/sec \
  --hashlimit-burst 20 \
  --hashlimit-name conn_rate_limit \
  --jump ACCEPT
iptables --append RATE-LIMIT --match limit --limit 1/sec --jump LOG --log-prefix "IPTables-Rejected: "
iptables --append RATE-LIMIT --jump REJECT

# ALLOW ESTABLISHED CONNECTIONS
iptables --append INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Cloudflare WAF:

Block Known Bots (expr: (cf.client.bot))
Block by Thread Score (expr: (cf.threat_score ge 5))
Block If Traffic Matching the URL http & https on any URL from the same IP address exceeds 5 requests per 10 seconds

Tomorrow evening I intend to go to the cybercrime investigation department, but in the next 24 hours I still want to try to solve my problem on my own (or with your help).

Any advice is welcome.

Best regards, Florin.

«13456710

Comments

  • Just some observations:

    • Using iptables, and with the overhead of conntrack on top isn't usually worth it as you're expending the same CPU power that you're trying to prevent. Cue @yoursunny talking about AF_XDP :tongue:
    • Also, when using iptables write rules for the attacks that you actually see, instead of writing generic rules that may or may not apply to your situation.
    • You could also just throw in Cloudflare's IUAM or managed challenge and call it a day.
  • FlorinMarianFlorinMarian Member, Patron Provider

    @stevewatson301 said:
    Just some observations:

    • Using iptables, and with the overhead of conntrack on top isn't usually worth it as you're expending the same CPU power that you're trying to prevent. Cue @yoursunny talking about AF_XDP :tongue:
    • Also, when using iptables write rules for the attacks that you actually see, instead of writing generic rules that may or may not apply to your situation.
    • You could also just throw in Cloudflare's IUAM or managed challenge and call it a day.

    Thanks for the suggestions but IUAM has been activated since yesterday, but without any effect.

  • AXYZEAXYZE Member
    edited June 14

    There's not enough useful info - for example what Cloudflare plan do you have?

    Start with:

    1. Allow Cloudflare IP ranges https://www.cloudflare.com/ips/ on port 443, if you need 22 then whitelist your IP and your other server IP if your IP changes (you can hop from one server to another and add your IP once again)
    2. Block EVERYTHING else in firewall
    3. Make more fine-grained rate limiting on Cloudflare side. Make it lowest possible, then test your website and increase limits if they are too low.

    Block Known Bots
    This is Google, Pinterest, Baidu etc. bots. You dont need to block it at all.

    Check where IPs that are used in attack come from. If they are non-residential like Hetzner, DigitalOcean you can just block whole ASN in Cloudflare WAF.
    https://support.cloudflare.com/hc/en-us/articles/217074967-Configuring-IP-Access-Rules

  • sandozsandoz Member

    Voxility provides Layer 7?
    Probably you are being targeted by L7 Attacks.

    Did you check if someone inside your network isn't using your servers to launch ddos attacks?

    I know some people which uses OVH servers to attack other OVH servers because is not well filtered and can impact them since is in the same network.

    Don't know if OVH did patch it or if they had implemented any solution.

    Other possibility is to use Voxility always ON..
    You can contact Javapipe.com they can help you with that. Very nice guys, and they know what they are doing.

  • FlorinMarianFlorinMarian Member, Patron Provider
    edited June 14

    @AXYZE said:
    There's not enough useful info - for example what Cloudflare plan do you have?

    Start with:

    1. Allow Cloudflare IP ranges https://www.cloudflare.com/ips/ on port 443, if you need 22 then whitelist your IP and your other server IP if your IP changes (you can hop from one server to another and add your IP once again)
    2. Block EVERYTHING else in firewall
    3. Make more fine-grained rate limiting on Cloudflare side. Make it lowest possible, then test your website and increase limits if they are too low.
    1. Already added
    for i in `curl https://www.cloudflare.com/ips-v4`; do iptables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done
    for i in `curl https://www.cloudflare.com/ips-v6`; do ip6tables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done
    
    # Avoid racking up billing/attacks
    # WARNING: If you get attacked and CloudFlare drops you, your site(s) will be unreachable. 
    iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP
    ip6tables -A INPUT -p tcp -m multiport --dports http,https -j DROP
    
    1. Attack is clearly on https://www.hazi.ro/index.php (extracted from Suricata)
    2. Already done

    Thank you!

  • In case it was not obvious to you, your origin IPs are publicly known. Try out what @AXYZE mentioned.

    But anyway, this could only be the beginning. Once your website is well secured, they could move onto attacking your subnets for your services. But ddos isn't free anyway, maybe they will eventually stop.

  • AXYZEAXYZE Member

    @FlorinMarian said:

    @AXYZE said:
    There's not enough useful info - for example what Cloudflare plan do you have?

    Start with:

    1. Allow Cloudflare IP ranges https://www.cloudflare.com/ips/ on port 443, if you need 22 then whitelist your IP and your other server IP if your IP changes (you can hop from one server to another and add your IP once again)
    2. Block EVERYTHING else in firewall
    3. Make more fine-grained rate limiting on Cloudflare side. Make it lowest possible, then test your website and increase limits if they are too low.
    1. Already added
    for i in `curl https://www.cloudflare.com/ips-v4`; do iptables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done
    for i in `curl https://www.cloudflare.com/ips-v6`; do ip6tables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done
    
    # Avoid racking up billing/attacks
    # WARNING: If you get attacked and CloudFlare drops you, your site(s) will be unreachable. 
    iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP
    ip6tables -A INPUT -p tcp -m multiport --dports http,https -j DROP
    
    1. Attack is clearly on https://www.hazi.ro/index.php (extracted from Suricata)
    2. Already done

    Thank you!

    I've edited my comment and added suggestion about blocking whole ASN on CF side. Try it :)

  • FlorinMarianFlorinMarian Member, Patron Provider

    @sandoz said:
    Voxility provides Layer 7?

    Yes, on the https protocol and I had to give them the public-private key pair to do that.

    Probably you are being targeted by L7 Attacks.

    Did you check if someone inside your network isn't using your servers to launch ddos attacks?

    Yes, new connections come from IP addresses around the world and we also tested all 3 locations: France, Germany and Romania - even if I had IP Spoofing locally, it would not have been in all 3 locations.

    I know some people which uses OVH servers to attack other OVH servers because is not well filtered and can impact them since is in the same network.

    Not applicable in Romania.

    Don't know if OVH did patch it or if they had implemented any solution.

    Other possibility is to use Voxility always ON..

    Already happen.

    You can contact Javapipe.com they can help you with that. Very nice guys, and they know what they are doing.

    Thanks !

  • FlorinMarianFlorinMarian Member, Patron Provider

    @NoComment said:
    In case it was not obvious to you, your origin IPs are publicly known. Try out what @AXYZE mentioned.

    But anyway, this could only be the beginning. Once your website is well secured, they could move onto attacking your subnets for your services. But ddos isn't free anyway, maybe they will eventually stop.

    It's weird when you fight and you don't know who

    Thanked by 1yoursunny
  • AXYZEAXYZE Member

    @sandoz said:
    I know some people which uses OVH servers to attack other OVH servers because is not well filtered and can impact them since is in the same network.

    Don't know if OVH did patch it or if they had implemented any solution.

    They didn't and won't filter it. Their 'VAC' works only on external traffic.
    This is probably cost-cutting on their part - they don't need to analyze internal traffic.

    Thanked by 1FlorinMarian
  • HotmarerHotmarer Member

    @FlorinMarian Did you just try to apologize to @tinyweasel? This helped with the LET attacks.

  • jackbjackb Member, Host Rep
    edited June 14

    iptables --append INPUT --protocol tcp --match conntrack --ctstate NEW --jump LOG --log-prefix "NEW TCP CONN: "

    Your problem here is you're probably going to hit disk io or CPU limits before network. This statement will kill your server, especially if it's spinning rust, because you're logging every single new connection through the kernel into a log file.

    I wouldn't be surprised if you're not even under attack. It's easy to take down your own server with iptables if you're not careful.

    @FlorinMarian try removing that statement, and I bet you'll see an improvement.

    I would also agree with @stevewatson301 that conntrack is probably unnecessary here. It won't cause any serious problems until you reach about 50-100kpps but if you want to get above 100kpps on a single server you'll need to disable conntrack.

    Thanked by 2Andreix Falzo
  • yoursunnyyoursunny Member, IPv6 Advocate

    Publish the IP addresses of the attacker and put them on the naughty list.

    Thanked by 3Hotmarer SinSiXX taizi
  • HxxxHxxx Member
    edited June 14

    path.net? try that
    @ChrisMiller

  • YmpkerYmpker Member

    Perhaps Sucuri would be a solution for your main website? https://sucuri.net/ddos-protection/

  • HxxxHxxx Member
    edited June 14

    Path.net i think it would help, seems to be a preferred solution that is used by many providers.

    Try moving your website to a provider that use path.net like Frantech, LevelOneServers, I'm sure there are others. Path.net runs tempest.

    You could also gre tunnel from Frantech.

    IMHO is easier to just use a VPS that use an IP already filtered.

    Good luck

  • AbdAbd Member, Patron Provider
    edited June 14

    you can also try reverse proxy through combahton with their layer7 protection. similar to path, they will need your certificate details.
    Edit: they call it flowShield

    Thanked by 2FlorinMarian Ympker
  • Peppery9Peppery9 Member
    edited June 14

    I opened your site, clicked a single link and found myself banned.

  • tjntjn Member
    edited June 14

    @Peppery9 said:
    I opened your site, clicked a single link and found myself banned.

    FYI - Same

    Hope it calms down for you soon @FlorinMarian

    Thanked by 1FlorinMarian
  • stefemanstefeman Member
    edited June 14

    Multiple ways to fix it..

    Enable hcaptcha from page rules.. The attacker is using proxies and privacypass to get to you. The bots cannot solve hcaptcha unless the attacker pays to east asian workers via 2captcha API for each pageview which is very unlikely.

    If its HTTPS 1.1 traffic, simply block that as its ancient and not neccessary unless the user is on android 2.x or uses Windows XP.

    Anti-DDoS rules in firewall section should be enabled at Cloudflare panel.

    Check Cloudflare panel firewall logs and add obvious ASN's/user agents to blocklist in Cloudflare. Nobody will browse ur website via scaleway IPs or via Bahrain hosting provider ASN's. Those are all proxies. Finally block IPs that have over 1% of requests during the attacks in CF logs.

    UAM does nothing since its easy to bypass JS challenges with bots and privacypass.

    Also get rid of those shit iptables. If you absolutely must use them, do it in prerouting chain rather than input chain and fix the order to solve the CPU issue.

    Also, why in the living fuck are you using REJECT instead of DROP at the bottom. REJECT responds and wastes your CPU.

    Not to mention your backend is basically public rather than masked behind cloudflare, so even when its finally blocked this guy will nuke it to orbit by flooding it directly as everything is exposed.

    Graph courtecy to DNSDumpster

    Then theres this, https://search.censys.io/search?resource=hosts&sort=RELEVANCE&per_page=25&virtual_hosts=EXCLUDE&q=hazi.ro

    And don't even get me started about whois history like domaintools.com which can list even brief A record changes over the years, which usually exposes the backend unless the first ever A record for the domain was added in Cloudflare control panel as masked entry.

    Catch the motherfucker and sue his ass. Nobody should need to go through this kind of bullshit at this day and age.

  • Otus9051Otus9051 Member

    Your thing is now on BleepingComputer
    https://www.bleepingcomputer.com/news/security/cloudflare-mitigates-record-breaking-https-ddos-attack/
    Quite not mitigated, author.

    Thanked by 2Frameworks Abd
  • Otus9051Otus9051 Member

    @Hxxx said:
    Path.net i think it would help, seems to be a preferred solution that is used by many providers.

    Try moving your website to a provider that use path.net like Frantech, LevelOneServers, I'm sure there are others. Path.net runs tempest.

    You could also gre tunnel from Frantech.

    IMHO is easier to just use a VPS that use an IP already filtered.

    Good luck

    Tempest doesn't have IPv6, coming soon in like 6-7 months.
    100UP is good too.

  • FlorinMarianFlorinMarian Member, Patron Provider

    @Otus9051 said:
    Your thing is now on BleepingComputer
    https://www.bleepingcomputer.com/news/security/cloudflare-mitigates-record-breaking-https-ddos-attack/
    Quite not mitigated, author.

    The graphs started to look better last night

  • szymonpszymonp Member

    @Otus9051 said:
    Your thing is now on BleepingComputer
    https://www.bleepingcomputer.com/news/security/cloudflare-mitigates-record-breaking-https-ddos-attack/
    Quite not mitigated, author.

    It doesn't mention OP anywhere in that article

  • SirFoxySirFoxy Member

    @stefeman said:
    Multiple ways to fix it..

    Enable hcaptcha from page rules.. The attacker is using proxies and privacypass to get to you. The bots cannot solve hcaptcha unless the attacker pays to east asian workers via 2captcha API for each pageview which is very unlikely.

    If its HTTPS 1.1 traffic, simply block that as its ancient and not neccessary unless the user is on android 2.x or uses Windows XP.

    Anti-DDoS rules in firewall section should be enabled at Cloudflare panel.

    Check Cloudflare panel firewall logs and add obvious ASN's/user agents to blocklist in Cloudflare. Nobody will browse ur website via scaleway IPs or via Bahrain hosting provider ASN's. Those are all proxies. Finally block IPs that have over 1% of requests during the attacks in CF logs.

    UAM does nothing since its easy to bypass JS challenges with bots and privacypass.

    Also get rid of those shit iptables. If you absolutely must use them, do it in prerouting chain rather than input chain and fix the order to solve the CPU issue.

    Also, why in the living fuck are you using REJECT instead of DROP at the bottom. REJECT responds and wastes your CPU.

    Not to mention your backend is basically public rather than masked behind cloudflare, so even when its finally blocked this guy will nuke it to orbit by flooding it directly as everything is exposed.

    Graph courtecy to DNSDumpster

    Then theres this, https://search.censys.io/search?resource=hosts&sort=RELEVANCE&per_page=25&virtual_hosts=EXCLUDE&q=hazi.ro

    And don't even get me started about whois history like domaintools.com which can list even brief A record changes over the years, which usually exposes the backend unless the first ever A record for the domain was added in Cloudflare control panel as masked entry.

    Catch the motherfucker and sue his ass. Nobody should need to go through this kind of bullshit at this day and age.

    mans drops some solid wisdom here and there

  • @FlorinMarian said: Any advice is welcome.

    I've faced dozens of attacks against my hobby projects. Different size. They are always has +- the same patterns.

    First, and the most important.

    1.) purchase x4b.net & some server where you will host your attacked service
    2.) purchase ANY 3rd party email provider which will NOT leak your server IP when sent registration / notification emails from this attacked website
    3.) add tunnel between x4b.net & your protected web-server
    4.) block all other connection
    5.) profit.

    Estimated costs: ~30 usd / mo

    That's what about fast solution which just works out the box without any problems.

    Second solution:
    Sucuri.net - for extra 10-20 usd / mo you can get +- the same thing in terms of proxy between your web-server. The highest risks comes when you leak your real web-server IP. In complex systems with many web-apps, email servers - easier to do.

    Third solution
    Requires a lot of time, and chances to survive under active ddos pretty low.

    Usually i do the same over and over again:
    1.) clean purchased web-server somewhere
    2.) added transparent proxy to web-port
    3.) In DNS A & other records - i write proxy IP
    4.) mx & mail-servers not related anyhow to web-host
    5.) to web-server added fastcgi_cache with ignoring some client paths depends on client authorization inside the panel.
    6.) optionally to nginx added hcaptcha module for showing captcha
    7.) crowdsec added too, which will over time block mostly whole botnet.
    8.) optionally you can grep banned IP's and send over ssh to your proxy server and block them with ufw / ipset / iptables i dunno what you're using like every 1 hour or so.

    About cloudflare:

    1. they does not block most of attacks since maybe i dunno what time.
    2. People who saying that they blocking ddos L7 on free / pro accounts, never faced any attack at all.
    3. CF useful for hidding IP & for CDN role, nothing more.

    About DDoS:
    my experience telling me, that the attacker - real guy, who emotionally a short time BEFORE were contacted or interacted with. It's can be a guy in support chat, in support tickets, terminated service, etc.

    Rarely: blackmail idiots.
    Or: ads campaign and dirty as heck competitor (close one) tried to do that

    Goodluck.

  • risharderisharde Member

    Wait, so cloudflare is useless for these attacks? I always thought my backup protection would be cloudflare if I ever got desperate.

    Thanked by 1Logano
  • stefemanstefeman Member

    @risharde said:
    Wait, so cloudflare is useless for these attacks? I always thought my backup protection would be cloudflare if I ever got desperate.

    Its not. It depends how you implement it.

    Thanked by 1risharde
  • voxility not good?

  • PulsedMediaPulsedMedia Member, Patron Provider

    What is the volume of the attack, how many requests per second?
    What are the CPU usage metrics?

    Which pages do they hit, any pattern there?
    Do they make full page loads, or just the single page?

Sign In or Register to comment.