All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Massive Layer7 attack, more than 33 hours
Good evening!
Since yesterday morning I have been facing a major problem related to the security of my website (whmcs), waking up suddenly with an attack coming from nowhere that does not stop.
From the information collected through CloudFlare / iptables / IDS, I have the following conclusions:
1. Attacks use tens of thousands of different IP addresses
2. The attacks come from almost every country in the world (I also saw IP addresses in my country, Romania)
3. The attack is so clever that it runs 24/7 (it resumes automatically when I turn off LiteSpeed for a certain amount of time)
4. The attack uses enough intensity to keep my processor 100% busy (I tested i7-6700K, Ryzen 3800X, 2x E5-2699 V3), all were sooner or later exhausted but without using for example 8000 connections if the processor took only 800 to overload.
5. CloudFlare rules both WAF and "Rate limiting rules" although they block thousands of IP addresses per minute, still pass enough IP addresses through their tests and reach the final target
6. OVH Game or Advanced 10Gbps does not detect the attack at all
7. Voxlity Layer 7 AntiDDoS blocks many IP addresses but still gets a few hundred connected in parallel and paralyzes the entire web server
Last iptables rules applied:
#!/bin/bash
modprobe ip_conntrack
/sbin/sysctl -w net.netfilter.nf_conntrack_max=4194304
echo "524288" > /sys/module/nf_conntrack/parameters/hashsize
# ACCEPT LOOPBACK
iptables --append INPUT -i lo -j ACCEPT
# FIRST PACKET HAS TO BE TCP SYN
iptables --append INPUT -p tcp ! --syn -m state --state NEW -j DROP
# DROP FRAGMENTS
iptables --append INPUT -f -j DROP
# DROP XMAS PACKETS
iptables --append INPUT -p tcp --tcp-flags ALL ALL -j DROP
# DROP NULL PACKETS
iptables --append INPUT -p tcp --tcp-flags ALL NONE -j DROP
# DROP EXCESSIVE TCP RST PACKETS
iptables --append INPUT -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/second --limit-burst 2 -j ACCEPT
# DROP ALL INVALID PACKETS
iptables --append INPUT -m state --state INVALID -j DROP
iptables --append FORWARD -m state --state INVALID -j DROP
iptables --append OUTPUT -m state --state INVALID -j DROP
# DROP RFC1918 PACKETS
iptables --append INPUT -s 10.0.0.0/8 -j DROP
iptables --append INPUT -s 172.16.0.0/12 -j DROP
iptables --append INPUT -s 192.168.0.0/16 -j DROP
# DROP SPOOFED PACKETS
iptables --append INPUT -s 169.254.0.0/16 -j DROP
iptables --append INPUT -s 127.0.0.0/8 -j DROP
iptables --append INPUT -s 224.0.0.0/4 -j DROP
iptables --append INPUT -d 224.0.0.0/4 -j DROP
iptables --append INPUT -s 240.0.0.0/5 -j DROP
iptables --append INPUT -d 240.0.0.0/5 -j DROP
iptables --append INPUT -s 0.0.0.0/8 -j DROP
iptables --append INPUT -d 0.0.0.0/8 -j DROP
iptables --append INPUT -d 239.255.255.0/24 -j DROP
iptables --append INPUT -d 255.255.255.255 -j DROP
# ICMP SMURF ATTACKS + RATE LIMIT THE REST
#iptables --append INPUT -p icmp --icmp-type address-mask-request -j DROP
#iptables --append INPUT -p icmp --icmp-type timestamp-request -j DROP
#iptables --append INPUT -p icmp --icmp-type router-solicitation -j DROP
#iptables --append INPUT -p icmp -m limit --limit 2/second -j ACCEPT
iptables --append INPUT -j REJECT -p icmp --icmp-type echo-request
# ACCEPT SSH
iptables --append INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT
# DROP SYN-FLOOD PACKETS
iptables --new-chain RATE-LIMIT
iptables --append INPUT --protocol tcp --match conntrack --ctstate NEW --jump LOG --log-prefix "NEW TCP CONN: "
iptables --append INPUT --match conntrack --ctstate NEW --jump RATE-LIMIT
iptables --append RATE-LIMIT \
--match hashlimit \
--hashlimit-mode srcip \
--hashlimit-upto 50/sec \
--hashlimit-burst 20 \
--hashlimit-name conn_rate_limit \
--jump ACCEPT
iptables --append RATE-LIMIT --match limit --limit 1/sec --jump LOG --log-prefix "IPTables-Rejected: "
iptables --append RATE-LIMIT --jump REJECT
# ALLOW ESTABLISHED CONNECTIONS
iptables --append INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
Cloudflare WAF:
Block Known Bots (expr: (cf.client.bot))
Block by Thread Score (expr: (cf.threat_score ge 5))
Block If Traffic Matching the URL http & https on any URL from the same IP address exceeds 5 requests per 10 seconds
Tomorrow evening I intend to go to the cybercrime investigation department, but in the next 24 hours I still want to try to solve my problem on my own (or with your help).
Any advice is welcome.
Best regards, Florin.
Comments
Just some observations:
Thanks for the suggestions but IUAM has been activated since yesterday, but without any effect.
There's not enough useful info - for example what Cloudflare plan do you have?
Start with:
Block Known Bots
This is Google, Pinterest, Baidu etc. bots. You dont need to block it at all.
Check where IPs that are used in attack come from. If they are non-residential like Hetzner, DigitalOcean you can just block whole ASN in Cloudflare WAF.
https://support.cloudflare.com/hc/en-us/articles/217074967-Configuring-IP-Access-Rules
Voxility provides Layer 7?
Probably you are being targeted by L7 Attacks.
Did you check if someone inside your network isn't using your servers to launch ddos attacks?
I know some people which uses OVH servers to attack other OVH servers because is not well filtered and can impact them since is in the same network.
Don't know if OVH did patch it or if they had implemented any solution.
Other possibility is to use Voxility always ON..
You can contact Javapipe.com they can help you with that. Very nice guys, and they know what they are doing.
Thank you!
In case it was not obvious to you, your origin IPs are publicly known. Try out what @AXYZE mentioned.
But anyway, this could only be the beginning. Once your website is well secured, they could move onto attacking your subnets for your services. But ddos isn't free anyway, maybe they will eventually stop.
I've edited my comment and added suggestion about blocking whole ASN on CF side. Try it
Yes, on the https protocol and I had to give them the public-private key pair to do that.
Yes, new connections come from IP addresses around the world and we also tested all 3 locations: France, Germany and Romania - even if I had IP Spoofing locally, it would not have been in all 3 locations.
Not applicable in Romania.
Already happen.
Thanks !
It's weird when you fight and you don't know who
They didn't and won't filter it. Their 'VAC' works only on external traffic.
This is probably cost-cutting on their part - they don't need to analyze internal traffic.
@FlorinMarian Did you just try to apologize to @tinyweasel? This helped with the LET attacks.
iptables --append INPUT --protocol tcp --match conntrack --ctstate NEW --jump LOG --log-prefix "NEW TCP CONN: "
Your problem here is you're probably going to hit disk io or CPU limits before network. This statement will kill your server, especially if it's spinning rust, because you're logging every single new connection through the kernel into a log file.
I wouldn't be surprised if you're not even under attack. It's easy to take down your own server with iptables if you're not careful.
@FlorinMarian try removing that statement, and I bet you'll see an improvement.
I would also agree with @stevewatson301 that conntrack is probably unnecessary here. It won't cause any serious problems until you reach about 50-100kpps but if you want to get above 100kpps on a single server you'll need to disable conntrack.
Publish the IP addresses of the attacker and put them on the naughty list.
path.net? try that
@ChrisMiller
Perhaps Sucuri would be a solution for your main website? https://sucuri.net/ddos-protection/
Path.net i think it would help, seems to be a preferred solution that is used by many providers.
Try moving your website to a provider that use path.net like Frantech, LevelOneServers, I'm sure there are others. Path.net runs tempest.
You could also gre tunnel from Frantech.
IMHO is easier to just use a VPS that use an IP already filtered.
Good luck
you can also try reverse proxy through combahton with their layer7 protection. similar to path, they will need your certificate details.
Edit: they call it flowShield
I opened your site, clicked a single link and found myself banned.
FYI - Same
Hope it calms down for you soon @FlorinMarian
Multiple ways to fix it..
Enable hcaptcha from page rules.. The attacker is using proxies and privacypass to get to you. The bots cannot solve hcaptcha unless the attacker pays to east asian workers via 2captcha API for each pageview which is very unlikely.
If its HTTPS 1.1 traffic, simply block that as its ancient and not neccessary unless the user is on android 2.x or uses Windows XP.
Anti-DDoS rules in firewall section should be enabled at Cloudflare panel.
Check Cloudflare panel firewall logs and add obvious ASN's/user agents to blocklist in Cloudflare. Nobody will browse ur website via scaleway IPs or via Bahrain hosting provider ASN's. Those are all proxies. Finally block IPs that have over 1% of requests during the attacks in CF logs.
UAM does nothing since its easy to bypass JS challenges with bots and privacypass.
Also get rid of those shit iptables. If you absolutely must use them, do it in prerouting chain rather than input chain and fix the order to solve the CPU issue.
Also, why in the living fuck are you using REJECT instead of DROP at the bottom. REJECT responds and wastes your CPU.
Not to mention your backend is basically public rather than masked behind cloudflare, so even when its finally blocked this guy will nuke it to orbit by flooding it directly as everything is exposed.
Graph courtecy to DNSDumpster
Then theres this, https://search.censys.io/search?resource=hosts&sort=RELEVANCE&per_page=25&virtual_hosts=EXCLUDE&q=hazi.ro
And don't even get me started about whois history like domaintools.com which can list even brief A record changes over the years, which usually exposes the backend unless the first ever A record for the domain was added in Cloudflare control panel as masked entry.
Catch the motherfucker and sue his ass. Nobody should need to go through this kind of bullshit at this day and age.
Your thing is now on BleepingComputer
https://www.bleepingcomputer.com/news/security/cloudflare-mitigates-record-breaking-https-ddos-attack/
Quite not mitigated, author.
Tempest doesn't have IPv6, coming soon in like 6-7 months.
100UP is good too.
The graphs started to look better last night
It doesn't mention OP anywhere in that article
mans drops some solid wisdom here and there
I've faced dozens of attacks against my hobby projects. Different size. They are always has +- the same patterns.
First, and the most important.
1.) purchase x4b.net & some server where you will host your attacked service
2.) purchase ANY 3rd party email provider which will NOT leak your server IP when sent registration / notification emails from this attacked website
3.) add tunnel between x4b.net & your protected web-server
4.) block all other connection
5.) profit.
Estimated costs: ~30 usd / mo
That's what about fast solution which just works out the box without any problems.
Second solution:
Sucuri.net - for extra 10-20 usd / mo you can get +- the same thing in terms of proxy between your web-server. The highest risks comes when you leak your real web-server IP. In complex systems with many web-apps, email servers - easier to do.
Third solution
Requires a lot of time, and chances to survive under active ddos pretty low.
Usually i do the same over and over again:
1.) clean purchased web-server somewhere
2.) added transparent proxy to web-port
3.) In DNS A & other records - i write proxy IP
4.) mx & mail-servers not related anyhow to web-host
5.) to web-server added fastcgi_cache with ignoring some client paths depends on client authorization inside the panel.
6.) optionally to nginx added hcaptcha module for showing captcha
7.) crowdsec added too, which will over time block mostly whole botnet.
8.) optionally you can grep banned IP's and send over ssh to your proxy server and block them with ufw / ipset / iptables i dunno what you're using like every 1 hour or so.
About cloudflare:
About DDoS:
my experience telling me, that the attacker - real guy, who emotionally a short time BEFORE were contacted or interacted with. It's can be a guy in support chat, in support tickets, terminated service, etc.
Rarely: blackmail idiots.
Or: ads campaign and dirty as heck competitor (close one) tried to do that
Goodluck.
Wait, so cloudflare is useless for these attacks? I always thought my backup protection would be cloudflare if I ever got desperate.
Its not. It depends how you implement it.
voxility not good?
What is the volume of the attack, how many requests per second?
What are the CPU usage metrics?
Which pages do they hit, any pattern there?
Do they make full page loads, or just the single page?