New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Massive Layer7 attack, more than 33 hours
This discussion has been closed.
Comments
Haha! Apologies, @MiguelM! 🤦♂️
Having spent way too much time in the business world I imagined your report as a pdf that you prepare for your clients.
The "page 7" reference threw me. Now I understand that you meant page 7 of this discussion thread. I imagined flipping pdf pages to page 7.
I even was gonna ask you if I could have the original source file rather than the pdf. But I decided not to make trouble.
Here's a link to an intrusion report I saw awhile back: https://github.com/security-union/going-fishing-with-my-raspberry-pi/blob/master/going_fishing_raspberry_pi.pdf
What kinds of reports do people like, for example, Path send to their clients?
Thanks, guys! Greetings from Sonora!
Hey @SplitIce ,
Important disclaimer: I'm sorry if I seem a little 'aggressive' in this text, the point here was to show everyone the behavior of a 4 Gen attack, and after reading it a couple of times, I went to the conclusion that it seems a little bit aggressive, I'm sorry for that!
Thanks for your reply, it will allow us to start a whole conversation about Layer7 Mitigation, right now we don't use reputation databases in order for DDoS Mitigation, so our mitigations are kinda different.
Well, we actually have a system that we like to call "Public Database" where all flagged IPs attacking customer's services, will be inserted into our databased and challenged all over our clients' services (of course there are many verifications behind this), the system itself is called "Public" due to the fact that we will make this database organized and public on our website, so anyone can fetch and simply block those IPs (Which is our own reputation database).
After what you wrote, I also saw that those IPs were already inserted into our database which means, those IPs were already flagged by our systems, and I would like to mention that I did reports in the past for those IPs.
Before starting, I would let you know that in this reply I'll work simply with real logs that I took some minutes ago (As I mentioned before, seems like this attack is being launched 24/7 and now, it's not still stopped), so here, I will not talking by speculation or what is 'expected by a mitigation', but with real logs.
Let me start by, our challenge just shows up in case of our Sensor Mode detects a Gen4 DDoS Attack, any other Generation will be flagged by the DPI systems, and there are things that whoever is reading this needs to understand that as with any javascript challenge, cookies are set, in our case, I'll talk about 4 specific cookies being them:
-> _dwt > Challenge token identifier, responsible for the challenge tokens to the server 'knows' the current challenge state for that connection.
-> _dwsession > Cookie generated at external API requests in order to the request get his token to be able to request further API Calls and proceed with his challenge identification and steps (This requires the browser to be able to understand the initial challenge verification and be able to request API Calls and process the response).
-> _dwcfp > When the whole fingerprint processing is done, you'll get your respective fingerprint cookie (This requires the browser to fully be able to fetch and understand the fingerprinting).
-> cptoken > Captcha token identifier, responsible for the captcha tokens so the serverside 'knows' what captcha is being solved
(There are more cookies that I could talk about as an example, but those ones are enough).
Now, let's look into some logs:
Log #1: https://pastebin.com/v4ErgBp8
-> What is important to see on this log? This IP seems to be requesting the challenge multiple times, with well-defined headers for his current browser, being it: Chrome, an updated version with the newer chrome headers: "sec-fetch-user", "sec-ch-ua", "sec-ch-ua-mobile", and others...
-> System detected this request and dropped it, as it is acting with weird behavior over the challenge, after that, got the same log from multiple IPs, but each IP did a simple request, seems like is not trying to 'harm' the website
Log #2: https://pastebin.com/9B39TJFu
-> After that, those requests stopped, and new requests with new headers were received, right now we can see "_dwt" cookie, seems like these requests were able to process the basic Javascript and first API Requests in order to generate his token.
-> Even tho, this request also has the "_dwsession" which means it was able to parse and create his session and he can proceed in the rest of the API Calls.
Log #3: https://pastebin.com/zMGKZa5s
-> After that, those requests stopped, seems like the connection is in a different state we have the "_dwcfp" token, which means the connection successfully could parse fingerprinting code and create his fingerprint token. (Do anyone believe that a non-browser emulator aka Gen3/2/1 could parse a whole fingerprint code?)
-> After that, fingerprint signals (set on server-side) detected it as gen4 and forced captcha on it, you are always forced to pass challenge before going to the captcha
-> Did the bot stop there? No, the connection also has the cookies "cptoken" and "cptenc", which means the bot was successfully redirected to captcha and tried to solve the captcha, which created his captcha token and captcha encryption, connection couldn't be procced here since he couldn't pass the captcha (but tried).
As I didn't want this to seem weird I picked up the logs from the hour that I wrote this text, and upload a little bit of the logs, so anyone can see them for themself: https://static.diamwall.com/cdn-cgi/report/hazi/gen4-log.txt
Important things to get from here: All those logs had a specific User Agent, seems like the Bots didn't try to lie, also all headers are currently set, which seems like a real Chrome behavior, also the Bots could parse every single JavaScript code, including it could fully fetch all the fingerprinting signals, seems like a Gen4 behavior, a really good one.
-> As we can see in what I call "Gen4 Logs" we don't see anything more than pure Chrome acting, so we can know that this Browser Emulator is currently using Chromium.
-> Those logs are separated from what systems consider to be Gen 1/2/3 or malformed requests, so the logs I'm about to show are from a different file
(I don't want anyone to feel weird about the last print's timing and the current timing, I had to go eat and I'll continue the text now).
When those Gen4 attacks occur they are mixed up WITH TONS of connections from another generation, I can give some examples:
Log #4: https://pastebin.com/FQEijfQ0
-> We can see what it claims to be a "Chrome" connection from a really, really old version with almost no headers on it (as obvious the version is really old)
Log #5: https://pastebin.com/pZCdF3B7
-> We can see what it claims to be a "Chrome" on a "Mac" with a more updated version, without the correct headers which should be sent my chrome, weird behavior (?), seems like a Gen 2/3 bot
As weird as it seems, I can tell that in minutes, I receive thousands of requests from Gen4, and millions from Gen 3/2, as I said in the past seems like the Gen 3/2 behavior is to eat the CPU or simply hide Gen4 connections.
I can also tell that, as obvious, most of these packets are malformed, empty, and with random user agents and obviously not able to bypass anything, so I quote again what I said on top.
(I'll not provide Gen 2/3 logs due to the fact that is currently a file with A LOT of GB, since is really generating TONS of requests, but if anyone requires I'll cut some logs)
Important things to get from here: There are millions (and soon billions) of logs from this, seems like a Gen 3 and 2 behavior, low r/s are not happening here I can flag a single IP trying to do more than 500r/s before getting flagged.
The headers and the logs tell the whole story about what is currently happening, this seems like digital forensics.
Far as the attacks keep running I can provide any further information about it, I also saved all of these logs in case anyone wants to know more!
(Also important: This doesn't expose our systems either it is relevant for an attacker, either is dangerous for us, don't worry, everything explained here is safe to know)
I hope everyone understands, and once again, thank you @SplitIce for starting this conversation, I'm sorry to write a bible, seems like I like to write reports!
Best Regards,
Miguel Miranda
Note I'm only trying to help (as best I can without revealing specifics).
Browser Fingerprinting is the term you are looking for to describe alot of what you are doing, and thats often a good aproach.
FYI there are headless chromium browsers that are pretty good. Many of the obvious mistakes made in previous large bonets (like WebGl not being compiled in) are no longer the case. You may be able to find some areas they arent perfect however and interaction options can be limited.
Note: I'm not trying to tell you whats in your attack you are dealing with. Attacks evolve and there are many botnets around with different capabilities.
FYI Please do consult with someone familiar regarding DNT and EU law (specifically GPDR) before doing that. For your own sake. Your actions might be compliant, they might not be (depending on what tracking and data collection occurs behind the scenes to produce that verdict) and what "organisation" (supplimentary info) you plan to release.
It is very trivial to change user agent. It takes more effort but is also relatively easy to use the exact same TLS ciphers etc. to mimic what popular browsers use.
It sounds like the biggest difference then between you and more well known services is that the bots aren't capable of solving your relatively unknown captchas.
Oh and @MiguelM I do like your generation classification. I probably wouldnt have seperated Gen 1 & Gen 2 (because absense of Javascript Cookie has been the bog standard mitigation method for the cheap vendors for years which does both). But 3 & 4 yes.
Though...
Well I actually thought the same, in a basic way, I separated Gen1 and Gen2 because:
We don't see many differences here, Gen2 is a little bit 'well-formed', but in the old times, the differences between Gen1 and Gen2 were massive, it was enough to bypass those protections that existed in the past.
This is the only reason I separated them.
Btw,
That meme really looks like me when I'm creating Firewall Rules and separating them for identification XD
Hey guys,
I also want to give you all a last update about the current situation, just for the ones who are feeling a little bit of curiosity.
The attack was mitigated (and never went throw) since we installed our proxy over hazi, since then, seems like the attacker tried a few more methods, including as you can see 2 comments mine ago, he really tried some 'smart' Gen4 methods.
Right now, he is launching 24/7 attacks against hazi (for some reason), so I flagged all the IPs that he is doing on his 24/7 attack (all doing low req/s) and inserted them at: https://static.diamwall.com/cdn-cgi/report/hazi/ips.txt
Anyone with the same issue or anyone who wants to do something with these IPs these last 539 IPs that I added are really important, due to the fact that they are being used for 24/7 attacks.
Well, I guess it's everything, anything that you wanna talk about, you can always PM me!
Best Regards to everyone!
Miguel Miranda
Well another problem I see here is that you guys are basically shining a spotlight all over yourselves, with and in this thread, sharing information that would be better off shared at a later point in time.
So if you don't want to be under the spotlight, maybe get out of it?
I'm sure some of you get what I'm trying to say ✌️
Hey @DP,
I see and I understand what you are saying!
To be honest I just really wanted to share the last IP Addresses that were launching the 24/7 attack!
Meanwhile, everything is fine now, and we can let this thread go into the deep ocean of lowendtalk.
Best Regards!
Wow, it's surprising to see DigitalOcean and ColoCrossing IPs in the bottom of that list!
(sarcasm)
As for a side note, I would advise the owner of the website to get some legal help before using third party services that publish user's personal information such as IP address.
Since the company is EU based, General Data Protection Regulation applies.
The company have the following obligations:
A range of sanctions, including suspension of activities and fines can be imposed if your company doesn't comply with EU law on data protection.
The Data Protection Authority may impose a monetary fine instead of, or in addition to, the reprimand and/or ban on processing.
IP address are not private information! Come on it is very public information that can be easily found. Humm your IP is being logged right now!
Hey @caii,
"An IP address in isolation is not personal data under the Data Protection Act, according to the Information Commissioner (ICO). But an IP address can become personal data when combined with other information or when used to build a profile of an individual, even if that individual's name is unknown."
"The Data Protection Act regulates the collection and use of personal data. If data is not personal data it is not caught by the Act – but it is not always obvious whether data is personal data or not. An IP address in isolation is not personal data because it is focused on a computer and not an individual. This reasoning was applied by the Hong Kong Privacy Commissioner in a complaint about Yahoo!'s disclosure of information about a journalist to Chinese authorities (Hong Kong clears Yahoo! of privacy breach over jailed journalist, OUT-LAW News, 15/03/2007). The Commissioner wrote in his report: "an IP address per se does not meet the definition of 'personal data'". "
We sum it to,
Far as we don't associate those with more personal information which can be used to identify an individual, we are working with the law, so!
Thank you for the advice and for your worry!
Best Regards!
Court of Justice of the European Union decided that IP addresses to be personal data (PII or Personally Identifiable Information) and therefore are regulated by GDPR.
Therefore, businesses must obtain website visitors’ consent before collecting such information.
Anonymizing visitors' IPs allows achieving GDPR compliance.
Hey @caii,
I did a little bit more research about the final decision regarding this theme by the "Court of Justice of the European Union" (CJEU), and the only topic I could find about a "relevant" decision was:
"The CJEU essentially decided that dynamic IP addresses collected by an online media service provider only constitute personal data if the possibility to combine the address with data necessary to identify the user of a website held by a third party (i.e. user’s internet service provider) constitutes a mean “likely reasonably to be used to identify” the individual." - https://www.twobirds.com/en/insights/2016/global/cjeu-decision-on-dynamic-ip-addresses-touches-fundamental-dp-law-questions
Which, of course, does not apply to our current state.
I also found more topics regarding the same theme, all with the same answers, you can learn more at: https://www.whitecase.com/publications/alert/court-confirms-ip-addresses-are-personal-data-some-cases
Best Regards!
The problem is, as soon as you are aggregating by IP, then you are forming a profile of that person, using the IP address as the representative PII.
You are literally creating this file as a means to identify people who you have determined are on your naughty list. It is covered by GDPR.
Thanks to @FlorinMarian for starting this thread and to @MiguelM for posting his analysis of the possible attack types as well as logs and ip address lists. Thanks also to everyone else who has contributed to the thread. Thanks to the people who sent me PMs with additional information. Best!
Tom
That corresponds roughly to the advice I was given.
The specifics of course vary. It also depends on if its reasonable and public information. For example a list of Search Engine crawler IPs or all IPs within an ASN is A-OK (no PII).
IANAL YMMV
According to your data I built this "top 100 ASN used in attack" graph. I hope I didn't fuck anything up setting up bulk IP-to-DNS lookup lol
Fullres / mobileusers see https://ibb.co/8DxTyVG
Raw data (& more ASNs) https://pastebin.com/bTcv4nC1
netcup at #85 with 53 IPs! It is interesting that its that high as they have very strict policies and not that popular. Looks like all of these machines used in attacks were just hacked/exploited?
Also curious about BlazingSeo, never heard about this company before, looks like they are doing scraping stuff / proxies. Its #1 attack source by a pretty huge margin. What do you guys think about blocking this ASN completely? Will it lead to any false positives or something?
I also got some L7 DDoS tries from the similiar IPs, Cloudflare rules blocked pretty much everything from it without any downtime, CF passed only around ~100req/s to the origin.
No signs of AI based Gen4 stuff.
Maybe this attack is abusing Proxys / TOR. That would maybe explain that so much from BlazingSEO. Dont think that all are hacked/exploited.
See Nutcup hosting alot of Relays and TOR
https://nusenu.github.io/OrNetStats/w/as_number/AS197540
And the Both IPv4 from us that where involved in the past, where customers Exit Nodes too.
But who knows. Maybe Diamwall guys know more about it
Upstreamed by CC so it's almost like CC makes up most of the attack volume lol, I've had them blocked forever with no negative effect.
Those top 15-20 ASNs are almost identical to ASN sources of L7 attacks that I was receiving earlier this month. It was easily blocked by CloudFlare with some manual changes, but I also block most datacenter networks by default.
This thread made it onto my bookmark list. Thank you to everyone posting suggestions and overall helping the community out.
So it could be the same botnet, some small changes could be related to time difference. Or it could be couple of botnets and main one is the same, others are different.
Curious if other people in hosting space saw same thing!
@Blazingfast_IO & @MiguelM your opinion would be great. Should other people just block ASNs such as blazingseo?
Blazingfast_IO are registered in Ukraine, so there are now very serious threats and they will be pressed for balls
[email protected]
blazingseo isn't blazingfast
@SpeedTest said:
Just because the names are similar doesn't mean you should not do your research first.
Sorry, I meant blazingseo there
Hey @MiguelM,
Just wondering why you decided to blast our sales email with a pitch trying to get us to obtain your protection?
It lines up near perfectly with a 160k r/s flood we ate today on our site too…