New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Massive Layer7 attack, more than 33 hours
This discussion has been closed.
Comments
not good. Not awful, but not good.
Up to 8000 new connections per second until my server froze.
100%
https://www.hazi.ro/index.php
Single page
Not as expected.
I've discovered an interesting thing. If I read correctly those metrics, he is spoofing IP addresses even from range 0.0.0.0/8 and they're blocked by iptables.
I completely disabled CloudFlare, it's crazy to lie that maybe it'll get rid of the attacks. I prefer to work with you on an alternative solution.
You disabled Cloudflare without making any changes as suggested, and still left those slow ass input chain rules active on iptables.
Not to mention, this is not a Cloudflare issue and they are likely targeting you directly into backend IP instead.
Where does it say that?
I only see 10.0.0.0/8 and 0.0.0.0/0 or "anywhere".
I've did some changes and at least for now, they're visible.
HTTP/HTTPS was banned for any IP excepting CF's subnets and there was no change.
With my current LiteSpeed/Iptables optimizations:
Maybe it would be good to discuss this in private perhaps?
The last thing you'd want is your enemy(-ies) to know your next move(s).
Just a thought.
I didn't mentioned what I did exactly for this reason
First thing you do:
Go to: /security/waf/firewall-rules
Rule 1:
(ip.geoip.country in {"RU" "UA" "BY" "CN" "TW" "HK" "PH" "MY" "KR" "VN" "TH" "IN" "ID" "SG" "PK" "BD" "JP" "EC" "CO" "CL" "BR" "AR" "PS" "IR" "KZ" "RS" "JO" "AE" "YE" "OM" "TN" "MA" "DZ" "LY" "EG" "SA" "IQ" "SY" "LB" "KH" "ZA" "MM" "SC" "NP" "MW" "KE" "SD" "SS" "GT" "SO" "NE" "NG" "GH" "UG" "SV" "GI" "HN" "MX" "BW" "MN" "CM" "BJ" "MV" "SL" "NI" "VE" "GN" "PG" "GQ" "GW" "DO" "PY" "SK" "PE" "AL" "MD" "BG" "HU" "MK" "GY" "BO" "PA" "LA" "GE" "AO" "TR" "AM" "RW" "CR" "SZ" "ML" "AF" "TM" "KG" "PL" "ZW" "RO" "CZ"})Action: Legacy Captcha
Rule 2:
(ip.geoip.asnum in {14061 11572 16276 208673 201089 15169 36352 48090 57043 24961 51167 44220 60117 208666 262254 43317 32875 60781 49392 202939 49349 49981 54290 44812 48874 60355 34224 202425 53667 64425 206898 136175 49335 8075 31034 62068 17831 24940 12876 30844 46475 200052 8560 43541 9009 29632 213349 20473 28753 49367 14576 198651 197922 24971 43350 61071 207810 29550 41256 30455 31577 81 12400 33387 31898 19994 49453 40065 62282 32475 3356 14618 25369 16509 40021 6830 29119 202709 22773 3352})Action: Legacy Captcha
Rule 3: (Optional) (Blocks Tor exit nodes)
(ip.geoip.country eq "T1")
Action: Legacy Captcha
Go to: /security/bots
Enable Bot Fight Mode
Go to: /rules
Make a rule with Browser Integrity Check and Cache Deception Armor enabled in that order.
Go to: /security/ddos (Optional if nothing else works)
Ruleset action: Legacy Captcha
Ruleset sensitivity: High
Finally go to: /security and set filter to 30 mins or less and start checking and refreshing whatever gets past still and try to find something that most passing requests have common, like IP, ASN, or User agent TLS Protocol version.
This should fix most of the issues you have right now unless the guy attacks directly into your IP with L4 flood.
Let me guess LS uses Recaptcha unless any other 3rd party security modules are involved.
idk about LS but CF Legacy Captcha is hCaptcha
Above is the last time I got DDoS by someone.
I used to be hit by l7 attacks all the time, just make sure your backend IP doesn't leak & everything on your internet-facing application is proxied through Cloudflare. It will automatically be mitigated for free, make sure to change your IP after you proxied all that is needed, Cloudflare won't magically mitigate a DDoS attack if the attacker knows your host machine ips.
This is not CPU usage stats.
This is:
or
First is from top second is from atop.
htop will do as well.
Need to know where the time is spend.
How did you measure this? Is that what apache or whatever httpd daemon you did?
Not possible, TCP/IP requires that both directions are reachable in order to open the connection and in order to try to load your index.php
Do you have any dynamic content there? If not, say just templating or something like that, you could "precompile" it as index.html and just present that. Huge overhead savings.
If you have dynamic content there, re-consider if you really need that. If so, can it be cached?
You have at least configuration issues, you are hitting the configured max workers. Light content pages modern hardware should be easily able to serve that ~6k peak req/s as shown by this.
Further you have no http connections what-so-ever, you could offset some load by loading static content via http, or even better, separate server for them. A low cost VPS is enough for static content, and your users will experience faster page loads then.
requests in processing figure could also reveal where you are bottlenecking.
Have you tuned keep alive etc or even disabled? 8000 new connections per second can be tough on linux kernel & available ports, so keep alive needs to be minimized. Same with conntrack.
Another option for an easy win is to rename it to something like index2.php and just have a very basic text-only HTML page that has a single explanation and a link to the renamed page.
You could even add a rule in apache to make it treat this as a static file rather than invoking PHP.
Their HTTP protection will help to some extent, it will block a lot of traffic, but it won't block all. I received a 300k requests/s attack through Cloudflare (as per the same mail that you got) a few days ago. And my origin servers got +- 7000 requests/s. They did block a lot, but still, enough might pass for it to become a problem without additional measures.
These are some suggestions that might help, when using Cloudflare:
1) Make sure that the pages that can be static, are static.
Cloudflare released their "Cloudflare Pages" product a few years ago, it's free static website hosting and allows for unlimited requests and bandwidth. You won't have to worry about a thing. Other service like Vercel, or Netlify also exist but I don't think that they are made to deal with attacks.
2) Challenge traffic using firewall rules on Cloudflare that you do not expect.
In the latest Cloudflare blogs about HTTP floods, and from the attacks that I've seen myself, it seems like most attacks are coming from data centers. For a client area for a hosting company, there is no need for such IPs to reach your site. So I think it's safe to challenge those.
3) Limit what URI / query / path / method's are allowed.
Sometimes those HTTP floods use HTTP POST requests (to bypass caching) but in most cases, there is no legitimate reason to do a POST request to the homepage for example. If you configure a firewall rule to do a challenge if someone does try a POST request, then that might help.
4) When using Cloudflare, you can perhaps give their "Tunnels" a try, they are free.
With Cloudflare tunnels, you don't need to open up any port, your server will make a connection to the two nearest Cloudflare data centers and use that connection. You can then use the firewall from the data center to drop any requests as soon as possible (before it hits the server), as others mentioned that you might be getting direct attacks. OVH's firewall won't block internal traffic, but it will stop a lot.
These might help generally:
5) Check how services communicate internally.
When you have multiple services (e.g. HaProxy -> Nginx) that communicate internally, make sure that you are using a method that can take on such requests. In this example, if you let Nginx listen on a port, and HaProxy connect to that port there will be some (not a lot, but still) overhead for the TCP connection, and you only have so many ports, if they are all used for connections, then that might be an issue.
A way to resolve that would be to use unix sockets for internal communication, but when using that, make sure that the file descriptor / open file limit is high enough.
These might help when not using Cloudflare:
6) Drop traffic as soon as possible
iptables can become a bottleneck when not used correctly. Drop traffic as soon as you can (e.g. in pre-routing), so you are not wasting any resources on your end. I think xdp is the best way to drop traffic, but I haven't tried it, only read about it.
7) Try to use your resources as efficiently as you can.
If one IP is sending you 200 requests/s, then you can be pretty sure that, that is not normal. Instead of letting LS handle it (Setting up a TCP connection, doing TLS, letting your webserver respond) I'd suggest you just block their IP using your firewall automatically (for a few hours or so). Fighting IPs on L4 is much less resource-intensive, no need to serve a web page to obvious abusers.
These are some providers that I know of that offer HTTP filters in case nothing helps:
x4b, JavaPipe, BlazingFast, Combahton/Fastpipe, Qbine (Serverius).
-Tim
We would gladly help you with this.
You can protect your site with > @Blazingfast_IO said:
BlazingFast will whipe all your ddos.
Thanks for recommending us @Hxxx !
@FlorinMarian We can most certainly assist you with this. I will send you a PM shortly.
No lmao
Nothing lmao about it.
nothing, lmao about it.
"When I grow up I wanna be like Path.net"
u try to enable allwasys on litespeed? and do some manual IP block? CF Have so many bypass
Shame list: someone from LET, after seeing that I blocked the external attacks, started to attack only with IPs from Romania
@HostSlick's IP is on the naughty list.
why is reliably only allowing residential ips so hard?
I know there are multiple proxy services that sell residential proxies, but assuming they aren't using those, it seems like blocking anything coming from a datacenter fixes like 99% of the problems.
Assuming they have ur ip address - is blocking said list via iptables or hosts/deny not feasible?
I just don't think anyone from a residential or mobile ip should ever see the word "challenge" or "captcha" or "bot" when interacting with your website.
I went to the OP's order page and got a page "verifying you aren't a bot". If the user is on a residential ip they should be whitelisted to do whatever they want until they are proven toxic.
I am sure I am wrong here - just don't understand how.
Great thread - thanks for all the insight.
All IPs listed above are romanian residential. They're enough to kill your webserver if you don't count requests per minute by URI. And I didn't.
If anyone is interested in researching, this is the list of IP addresses and the number of related hits in first 6 hours of attack (hits are counted per URI, not per resource).
https://pastebin.com/4U1pJes5
Another one that belongs to @HostSlick.
Well, you have made it clear that you want to try to handle the attack on your server and not rely on external services. Maybe you should try out crowdsec which has a database of known botnet IPs.
so "ip add route blackhole xxx" doesn't help?
CF being hit a by lots of attacks these days. This was just recently and concerning a user on the free plan: