New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Massive Layer7 attack, more than 33 hours
This discussion has been closed.
Comments
@DP advised you something as far as I remember.
Curious here, how many ips are you being attacked from? Just wondering why iptable lists causes such a cpu spike. Amazing to know computers can still be controlled in the masses like this with all the fancy av products out there but maybe there is a need for a better Linux av product (I did make some assumptions here, no need to truly lecture me here, just too tired to type out all scenarios and focusing on perhaps exploits/rootkits)
This two IPV4 belong to two dedicated servers. The customer in question seems to be running Tor Exit Nodes.
I just talked with him and he will investigate and make sure the problem will be solved!
There's also proxies etc. and don't forget compromised routers and/or smart/iot devices. Antivirus wouldn't help with computers/servers if people are installing pirated/nulled software or running untrustworthy scripts anyway. But I would think it's mostly the routers/iot devices that have some known vulnerability and all of a sudden there's hundreds of thousands or even millions of new devices added to a botnet thanks to a single vulnerability.
https://github.com/dbContext/SiteShield-OpenResty
Disclaimer: I created it, but it's an open source LUA module built upon OpenResty (Nginx Fork) that challenges all requests via a JS challenge (that can be invisible) but can't as easily be manipulated / bypassed as it stores the "cookie" server side via redis instead of allowing the client to set a cookie.
This ipset and iptables --match-set combo looks much better than a long list of rules. I'll have to try this out!
This is old known issue for iptables. That's why exists ipset.
Good article can be found here: https://kinvolk.io/blog/2020/09/performance-benchmark-analysis-of-egress-filtering-on-linux/
To be honest, there are also different queue mechanisms in kernel, but about such things there are more educated guys than me which can tell more and better.
In short:
All of that can dramatically help to tank volumetric attacks on not bad dedi server.
This is not about linux hosts.
This is about cameras & routers.
Vendors after selling them - does not care about their responsibility for selling extra bots in 1 year. Why I think so? I rarely seen a router or IP camera with firmware updates AFTER 1 year of releasing the device which is purchased and installed for a long term (decade+).
Most of the IP cameras - one of the most insecure segment which is from hackers points of view - ideal target for launching massive ddos / bruteforce and other attacks because of the BW (channel) & hardware.
No firmware updates, more devices every year, no security patches, awful documentation, awful setup, clean ips (not even NAT) = boom we have ready for using part of botnet with 100mbit/s + channel + 128MB ram+ and few cores.
And I do not talk about extremely popular manufactures of home routers, which EOL in 1 year, used by millions of people, and has enabled ALL extremely insecure options out the box, even if CVE not published.
This is long story.
Every single year - more and more devices.
More goverments purchase 300.000+ cameras per year (numbers not a joke).
After 1 year such camera model = become a zombie.
Which Cloudflare plan are you on? If you're on a business plan or higher then you should be able to live chat to them about it and see what they recommend.
iptables is legacy and you really should be using nftables these days.
So thats why I had so much requests last week :Sadeg:
Attack still ongoing but I have a new fresh list (IPs banned last 10 minutes): https://pastebin.com/PEjk1i2E
I can tell you the following:
1. There are at least 3 attackers / attack methods used
2. Their IP addresses will be permanently banned and from my checks on abusedb.com, all including .gov (WTF) have been involved in web attacks.
Discovered methods of attack:
My server is still down but I'm doing my best not to let that happen.
Since I've already lost a lot of money, it doesn't affect me for a few more days as I gain knowledge.
>
please tl;dr, so they are attacking your server directly? not through cloudflare?
If you will read the document: https://media.hotnews.ro/media_server1/document-2022-05-2-25531225-0-raport-adrese-dnsc.pdf you will find out, that this botnet actively used against energy sector and against governments and countries, and cyberpolice guys from Romania interconnect the botnet with russians, which actively use it against everyone.
@FlorinMarian thank you for providing these details, sorry that you still under attack, did you try the suggestion of using x4b? Or contact cloudflare support? If none of these worked, sounds like your ip is somehow being leaked? And if so, have you checked to ensure you don't have a compromised server? Time to check all bases or even redirect that attack to something offsite even until you audit your server just in case?
I am no professional in this field of course but I do have flood attacks built into my server software which I don't think I have tested and thus your experience is invaluable as to thinking up even more solutions.
Can you block all ips except cloudflare ips? Shouldn't that solve the attack or am I missing something important here? @FlorinMarian
yes but traffic still goes to the server right? just stopped at the "frontdoor"
I disabled CloudFlare for 24 hours because it was useless.> @risharde said:
I don't use CloudFlare anymore because even though it blocked millions of requests, thousands of IP addresses were still enough to protect them to put my server in my head.
I know for sure that the attack is Layer7 because multiple requests are sent to the site index and then the connection is closed.> @risharde said:
I did this 2 days ago, it was totally useless.
I still think you should try something like this:
Your server should then be invisible to the attacker, and then you can point cloudflare at the VM.
but assuming they aren't using those, it seems like blocking anything coming from a datacenter fixes like 99% of the problems.
Suppose it depends on how much money you care to lose on falsely identified IPs.
An active attack is different from 'normal' firewall rules though, or should be..
@risharde said:
I think the problem is after blocking all IPs except CF, traffic from other IPs were hitting his servers because backend IPs are exposed. Either that or the traffic from CF IPs alone was high enough to take down his servers. I could be wrong though.
Of course, if they still overwhelm your server, then the cloudflare error message will still leak your IP address. I think they should address that.
Hosting loli websites and dosing other people?
It is tinyweasel, just apologize to him and it ends quickly.
If 8000 requests/sec to just your site index is freezing your server, that is the problem.
The point of external anti-ddos services like cloudflare or anything else is to offload the scrubbing and handling of some known attack vectors to the external service. If that doesn't work, then you have to investigate what exactly brings you down, and patch that. It's not exactly useless because it probably costed your attackers more time and money to bypass cloudflare. Go cache your site index or whatever and see what type of requests your attackers make next. Then patch those. They will eventually give up.
How can false identification be when you make abnormally many requests?> @jmaxwell said:
When you access the IP address of the backend you get a blank html page, not the whmcs content.
The attack is 100% Layer7, I clarified this a long time ago through access_log.> @NoComment said:
index.php queries the database. It's not like if I changed this behavior the attacker couldn't move the attack to cart.php or another more problematic page than that
TinyWeasel DDoS Service ᵀᴹ
But maybe we need generic solutions for everyone?
It makes a huge difference because some work has to be done to interact with your billing panel. Maybe the attacker can't use some stressers, maybe he has to run headless browsers and interact with stuff.
Sure enough, some of the things people have talked about here are somewhat subjective. If you think anti-ddos services are useless, that's totally okay. But 8000/sec get requests to your index should not freeze your server. This is 100% objective.
I guess people wasted their time writing essays here to help you out.
That can come later, when the issue is resolved
I did not say that Anti DDoS services are useless, I said that tested solutions were useless.
IUAM can be bypass