New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Massive Layer7 attack, more than 33 hours
This discussion has been closed.
Comments
From what I noticed, I have a problem with the threads created by OpenLiteSpeed, namely they are active even after 20 minutes even if the IP address that made the requests has been banned in the meantime by iptables.
I will have to figure out how to remove these threads in a much shorter time as it consumes a lot of CPU.
https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:mitigating-ddos-attacks#use_max_request_response_settings
Shorten timeouts for both connection and keep-alive, that should work.
I would give you perfect config for nginx, but my knowledge about OLS is quite limited. In nginx you can configure how many connections can go to worker and when to terminate old workers/threads & much more finegrain.
I'd be very interested in such a config!
If anything, I've found this thread super interesting and educational.
I would need to create completely different thread in order to explain all of this, because even "perfect config" needs to be customized to website
I will share most important piece - if you block everything but Cloudflare IPs your website will work fine. The problem is that you see traffic only from these Cloudflare IPs and thus you cant rate limit or have extra L7 protection per IP as it will affect both attackers and legit people.
BUT!
There's solution for that!
Cloudflare sends extra headers, we can use "CF-Connecting-IP" header to get real IP of visitors!
https://github.com/ergin/nginx-cloudflare-real-ip
That way you can make ratelimiting zones per IP and allow just legit numbers of requests to be passed. You can also drop connections to IP that consistently hit rate limits. It is very powerful against most L7 attacks & completly automated & no falsepositive if you SETUP CORRECTLY (test on legit users and do stupid stuff yourself, make sure you dont hit any rate limits that way)
It doesn't end there! There's also some other nice headers like "HTTP_CF_IPCOUNTRY" which uses Cloudflare GeoIP. If you have website that serves visitors mainly from one or several countries this if very powerful tool. You can make custom L7 protection that wont have bypass like CF methods of verification, but in the same time whitelist countries of your visitors.
Ok, one more thing. Microcaching is most underrated feature of nginx. Even if you cache something for like 2seconds it will help A LOT in case of attacks, because if someone is making 2000rq/s you can get answer to them instantly without hitting your database/PHP/whatever.
These things wont make your server bulletproof, because there's a lot more things to do, but it gets you very close!
... one more thing... okay?
If you serve only one country (lets say you have ecommerce), but you have attacks outside there's one extra way that nobody talks about. You can get two servers and route 100% of one country traffic to first, and then everything else to second. You can do it via Cloudflare Load Balacing (paid, need geobased load balancing extra which will total like $20/mo IIRC) or via Cloudflare Workers (very cheap or free if you dont have many requests per month). This is unusual way, most people just block other countries (not ideal, in case of ecommerce there's people abroad that want to buy something for someone else). This will make sure that your website stays online for 99%+ of your visitors and attackers will stop attack very fast because you are not affected by their attack and they just waste time. 
Seems that Racknerd's DDoS protection has protected my server from a DDoS attack yesterday, but while the DDoS protection was active, the server was re-routed from Multacom to Voxility. I found only TCP connection were forwarded and all UDP & ICMP packets were dropped. While my web services survived the attack, all service using UDP such as game server went offline, which is sad. Is there any solutions for protecting UDP service> @DP said:
Okay, I've started a new thread.
This should be done regardless of whether there is a DDOS attack, at least for statistics
Yes and no. Yes, iptables is legacy but No, as a professional provider one should not use [current linux firewall] but rather two dedicated firewalls, preferably with hardware support.
"https everywhere!!!" is what I suspect. If I'm not mistaken @FlorinMarian even said that the attacks (at least many) just establish a connection and then walk away. That's an attack vector we were bound to see and more often, because while relatively expensive and highly likely too hard for most IoT stuff (unless one is a real pro and knows how to forge good enough first packets) it is attractive because surprisingly few requests/packets are enough to take even beefy servers out of the game.
... which can often be achieved with a smart network adapter too, but alas most people incl. providers just use whatever happens to be in the box.
what a monumental BS
yeah, sure, our Romanian champion who used to build his servers from scraps bought from cociu level sources and other 3rd hand junk (ancient 9 years old CPUs, 6 years old retired HDDs, consumer grade ancient EVO SSDs etc.)... suddenly starts buying some nVidia (ex Mellanox) BlueField-3 DPU cards to implement anti DDOS filtering on them instead of using external widely available el cheapo services (which works very well for others), because his crap (HW specs and lack of config optimization) servers cannot handle 8000rps
if you want to implement anti DDOS on expensive smart NICs then check their prices first and then confront it with the budget of this "wanna be provider" kid (or maybe it was "when I grow up I want to be a real provider"?)
Mellanox 100Gbps is $800/unit, 200Gbps is $1200/unit; if you need optical transceiver, it's extra.
These can be configured to filter by IP + port, but cannot further recognize protocols.
BlueField is basically a tiny computer in the NIC, with its own ARMv8 processor and DRAM; I don't know how much they cost.
It takes more effort to program, but it can do much more.
You can even do TLS termination and serve static content inside the NIC.
Suppose you are hit by 1Tbps attack, you can absorb it with ten BlueField adapters.
I agree with your intro to your comment.
Do you have any credible evidence for that? I guess no, but I'm ready to look at any credible evidence you might provide.
There are others too
Totally different things.
One can get some for about $500 which shouldn't be out of reach for most LET providers.
And yes, it does make sense. For one the bandwidth those adapters can handle is far beyond what the typical LET provider has. But it's of course also helpful to have an extra (and usually more or less specific) hardware/"CPU" to relieve the systems main processor. Third it allows for a reasonable degree of independence from service providers like e.g. CF.
Yea, programming extremely hard thing to do. But there are a lot of good examples from where to start, like: https://github.com/pavel-odintsov/fastnetmon (i've seen in action such things on many ISP, pretty dope for software)
he many times posted announcements about his archeo purchases, so you could check details about his CPUs (i.e. on Intel ARK website there is release date Q3/2013), or look for his photos of hard drives (i.e. manufacture date 2015), or check for yourself what is Samsung 860 EVO SSDs dedicated for
there you have example of his purchase from last December:
https://lowendtalk.com/discussion/comment/3334904/#Comment_3334904
FastNetMon appears to be an Intrusion Detection System.
It's not designed to run on BlueField.
BlueField can be programmed with Data Plane Development Kit (DPDK).
I have programmed DPDK, but not on a BlueField.
It's possible to perform advanced filtering that understands protocol semantics, including TLS termination and filtering by URI.
@yoursunny do you really think, than he has servers with 10 PCI-e x16 slots, able to accommodate ten DPU cards (each costing thousands of dollars)? and 1Tbps uplink able to send all this traffic to his server? I don't think so
or looking how he is dealing with this current DDOS mess, do you think that programming these DPU would be easier for him?
point is, that is it much more effective to filter this DDOS traffic as much as possible at external services (i.e. such CF has network connectivity millions times better then single provider can imagine) instead of building such monstrous (10 DPUs) machines with beefy uplinks in your own (especially from el cheapo Romanian boy's perspective)
DPUs are great products, but they are dedicated for hyperscalers, not wanna be el cheapo starters... they would never earn for themselves in $14/y LET boxes
I can't find a server with 10 PCIe x16 slots.
EPYC 7003 series processor has 128 lanes, but there's no motherboard with 8 x16 slots.
The maximum I found among single EPYC configuration is 5 x16 slots and 2 x8 slots, because some lanes have to be reserved for NVMe, USB, etc.
Dual Xeon configuration can have 8 x16 slots, but they are behind 4 PCI switches.
Having PCI switches reduce available bandwidth.
I also heard complaints that they cause multiple NICs to be placed into the same IOMMU group, which causes problem with PCI passthrough via SR-IOV.
Buy 1Tbps Ethernet adapters and give each VPS dedicated 25Gbps port.
The Romanian boy will instantly become the best provider.
Ok so you disabled CF...
and are not testing any other antiDDoS.
This is not the way.
I believe you would have solved this by just buying a VPS with IP's filtered (probably that uses path net) move your site there, make the A record changes, done.
Other option was to get a new IP assigned to the server. Setup Cloudflare paid version, make sure your origin don't leak, and that should improve your situation.
imagine you just bought such 25Gbps VPS (for Romanian price)
for such high net spec it would be an involuntary reflex (at least for me) to run yabs. right?
and boom... you triggered super duper hiper Romanian anti evil protection... and you win 364 days on involuntary idling I mean service termination
fully agree
The SmartNIC can recognize and respond to iperf3 traffic locally.
You'll see 25Gbps without using Internet egress.
I saw that to many Providers here offered him to help , but he is ignoring all of them for some reason , and he showed that he is not capable to deal with ddos attacks so i think that he messed with someone that he should not do that, And thats the reason why he is being a target of ddos. Dude dont make enemies but make money and accept helps from other providers. I know a lot providers here that can stop L7 Attacks easy.
The issue is, that his backend is all public, so even if he fixed it by handing all keys and passwords to a competent helper on these forums, (which by the way is absolutely the most retarded idea ever, if you are a commercial provider with your own customers) the attacker could still just nuke his real IPs and bypass the Cloudflare protection.
Originally this guy became provider as a student project, so its fine to be incompetent, but if he specializes in anti-ddos as he claimed before, i've seen more competent discord based kid hosts with 14yo admins.
So?
You want a cheap VPS - you'll end up with old processors. You want modern processors - no problem, but you'll pay significantly more. What's the problem in your eyes?
Why 10 x16 PCIe slots when the total bandwith a provider has isn't even near 1 Tb/s? Here on LET we're more talking like n x 10 Gb or maybe, just maybe a few 100 Gb/s, plus the adapter @yoursunny mentioned isn't the only one, there are also quite some 2x25, 2x40, and up to 2x200 Gb/s adapters available, some of which even have a firewall built in; no real programming needed. Granted, those don't terminate TLS (but they usually do AES) but then that's an entirely different story anyway.
TL;DR for what providers like @FlorinMarian (and many others here) need there are adapters in the $500 price range available and those adapters, used as a firewall, can take a ton of load from the servers.
What he said
That's not true because he is colocating servers 1-2U at a time and he has 1 Tbps voxility anti-ddos so I'm guessing it comes as part of the colocation package or some kind of addon service. The networking is probably handled by the datacenter anyway for these 1-2U colocations.
Maybe when he is colocating an entire rack, it would make sense to consider scrubbing himself but at that point, why even bother with these adapters? In fact, with just one rack, maybe you would still be better off relying on someone else within the datacenter...
I hope you're kidding
i mean, now he is on path and it's still dead.
Hazi.ro when I enter your website is shows a document to be downloaded through mobile... What a hell.. Both https or http asks to download a file....
As if "[some] Anti-DDOS 1 Tb/s" meant that a provider really has a 1 Tb/s pipe ...
I think we both know that extremely few providers here at LET have that kind of bandwidth but hey, if e.g. Voxility has a 1 Tb/s connectivity in the DC (or even just via a MAN linking multiple DCs) even a provider with a single 10 Gb/s line can - and more often than not will - talk about "1 TB/s" protection.
I already answered that. It can make sense with pretty much any bandwidth.
The point was never the number, but anyway the 1 Tb/s probably would only last a few minutes/seconds anyway. If there was a 1 Tb/s attack that lasts 1 minute and they can filter it to 10 Gb/s of clean traffic, then that will work just fine.
My point is that at 1-2U colocation, everything networking related is already being dealt with for you.
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?
I am not. Literally go check his introduction thread and his first sales thread where he tells about it and his goals.
We made additional adjustments for him. Everything looks good now.