All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Distributed scraping attack from random IP's
hostcurator
Member
On June 25 2026 one of our magento production website was found having heavy load and website stopped working. I have checked the case and I could see it was heavy traffic to website and load has induced. Website is already behiend cloudflare proxy and we already have rate limitations via nginx for bots. When the issue happened, I initially enabled Nginx rate limiting and created a new Fail2Ban jail to automatically block abusive IPs. While monitoring the logs, I noticed the server was still under heavy load and PHP-FPM was repeatedly hitting its limits, eventually causing the website to return server errors.
After investigating further, it became clear that this wasn't a typical attack from a handful of IPs. It was a distributed scraping campaign where each IP was making only one or two requests, making traditional Nginx rate limiting and Fail2Ban largely ineffective since neither IP exceeded the configured thresholds.
Most of the requests were targeting Magento category and product pages with various filter combinations in the query string. These requests appeared to bypass Varnish caching, resulting in PHP generating the pages repeatedly, which explains why PHP-FPM became overloaded.
While analyzing the access logs more closely, I noticed another interesting pattern. A large number of these requests were using randomly generated, but consistently old, Windows and Chrome user agents. Based on that observation, I created a Cloudflare WAF custom rule to apply a Managed Challenge to requests matching those characteristics. Within just a few minutes, the rule had already triggered around 1,360 challenges, confirming that a significant amount of the traffic matched the pattern.
Since enabling the WAF rule, the server load has returned to normal, PHP-FPM is no longer being overwhelmed, and both the website and internal links are functioning normally again.
I'm still interested in understanding the root cause, though. Has anyone else encountered this type of distributed scraping against Magento, where thousands of IPs each make only a couple of requests specifically to filtered category/product pages? If so, how did you handle it without affecting legitimate users? I'm particularly interested in improving cache efficiency or identifying better ways to stop this type of traffic before it reaches the origin server. After the raise of AI, I am experiencing these kind of issues with random servers.
Comments
I notice a lack of anubis, stopped my repos getting hammered
Given i don't do a lot of webhosting these days i haven't but i think it'll become more and more common. It's just logical. Protection is being ramped up and the scrapers follow suit. Consider yourself lucky they gave you such an easy out with the outdated agent strings. Sadly that's very, very easy to fix on the scrapers side and the more these oversights lead to successful blocks the more they'll be avoided.
To be brutally honest i don't have a positive outlook as far as a functional web is concerned. As things currently stand there's basically no way around turning it into a wasteland of increasingly convoluted captchas and even that might not translate into a solution.
Hi,
essentially you have the choice between:
A ) paid protection ( akamai, cloudflare, ... )
B ) (higher) risk of false positive by going into GEO blocking, Timing blocking, ASN / IP Blocking, ... and others that will lead to 90% of attacker will be filtered out while 10% of legit users will be filtered out too. Means you have to deal with 10% attacker load and have still 90% of legit user traffic
Both is obviously annoying but thats how it is.
Depending on your location, in general i suggest to block all public clouds, all hosting type IPs, all VPN type of IPs and so on.
There is usually no real reason that any hosting/cloud IPs should access a webshop. ( But to scrap information ).
Also VPN is questionable if its required. But that depends on your own location and target market.
The times of free internet are already long gone. And with improving bots / AI / ... everything but IP / ASN / Network based filtering will soon turn completely ineffective. You have it already now that captcha's are loosing effectiveness. New captcha comes out, helps for some time and then loose again effectiveness. Endless circle.
So best is, filter GEO / IP Type / ASN based. Not nice. But all in all lesser evil.
Unfortunately, this is a pretty major thing that I'm not sure any one person has a good answer for.
Mythic Beasts has encountered the same sort of behavior: https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/
98% of bad actors rely on public proxies, lot of public projects on github allow you to retreive the majority of these every X interval set by you, apply these proxies to an auto-drop list either server-level or more ideal, cloudflare level leveraging there rules.
Stops craping, most simple sign-up ddos platforms etc.
Truth is, it’s a chasing game and it depends how much you want to suffer GEO blocking does nothing sadly, cheap rotating proxies are very accessible.
Platforms like akamai are great, truly great but everything is bypassable sneaker bots have been selling bypasses to cloudflare, fast, akamai for years for few $100/month
Well, being able to concentrate my browsing on a fixed (DC) IP has served me well over the years keeping Paypal from freaking out while staying abroad but, yeah, chances are that's coming to an end. Paypal has already become way more aggressive lately and i wouldn't be surprised at all if one day they'd just tell me to come back with a normal IP.
Hi,
yes, unfortunately at latest with broad availability of AI, developing something to get around a newly implemented filter becomes faster and easier than ever.
Soon the advantage / sense of "inventing" new filters becomes senseless if workarounds are available within hours and days.
Maybe in the future, you can add a specific IP to be whitelisted with big websites, if you are a paying / identified customer there and having this way access to the service also with a VPN IP.
Lets see where the road goes....
Assign an ipv6 /128 for a paying customer's panel/api access.
I have been seeing this for maybe a year now on a wiki-based web site where each IP requests a page with unique request parameters. It doesn't really look like the intent is to scrape anything, but just to overload the site.
What I noticed is that the user agents claim to be (old) Mozilla/Chrome web browsers, but they only speak HTTP/1.1, so I am now just giving them an HTML page with a meta refresh tag and they never come back to refresh. Real users would either use HTTP/2.0 or not claim to use Mozilla/Chrome - and in the unlikely event where they get caught, they would just see a small delay on the first load.
Using Cloudflare honestly seems like the easiest way to handle this.
Which plan are you on right now?
If you're on the Enterprise plan, they have a feature that lets you block all the proxy IPs that Cloudflare has collected.
I have been running CrowdSec and it works fine for removing quite a lot of them even on the free plan. Chain it with Anubis if you can't use Cloudflare to remove the endless bots. Unfortunately these bots are basically endless, every random website I host has had way more bot requests in the last few years, and there is no real good solution, fail2ban is not enough anymore but there's no real way to filter them out for now without adding captchas.
Not only that, men, but the bots, scrapers, and so-called AI will read these very comments of yours and, quote-unquote, learn and adapt. You are now food for bot thought.
Mentally strong people keep the websites well optimized to handle the workload from both humans and bots.
We are open to any and all bots for all intents and purposes:
This.
AI companies happily ignore
robots.txt. They just want your website's data and don't care if your servers are overloaded.For instance, Fedora, openSUSE and FreeBSD have Anibus now.
Well, you probably aren't too far off with that but then it's not like my comment is exactly new information. Ancient-agent-strings-will-make-you-stand-out is a pretty obvious mantra and someone/something ignoring it is more of a testament to their stupidity than some kind of advanced wisdom on my side. What browsers generally make the best spoofing targets would probably be at least a little less well known but even there anyone with half a brain and an interest has already implemented this ages ago.
Anime!
AI companies are purchasing data from datasupplierbroker, which is offshoot of ads, insurance, etc databroker which already operates on PB-scale 10 years ago.
Most AI lab are transparent about their user-agent. It's the human-user who decides to spoof AI-enabled retrieval.
I wonder when will the AI bros and the Bots fck up Anubis too
Well, if its a simple scrape, Anubis will cock block them with a js challenge
However, if you go full browser aka Selenium or similar, you can just bypass it.
Costs more resources though, might not feasible.
You could also try, to add a js engine, detect if a challenge is presented and run it with the js engine to solve it.
Anyway, more work, more resources.
I think something inspired in the old progressive JPEG technique, only now for content, could work, but it has to be done at application level and with a twist.
On first visit start by showing a "low-resolution" preview of your content to anyone who accesses your site and then impose some delay. During that delay run some checks to see if you are dealing with a bot or a human. Humans will be happily engaged with the content preview and tolerate the delay well but bots will not be "happy." After the delay deliver the full content and mark the users who passed your checks and tolerated the wait by giving them a fingerprint that's included in the session data. Use HMAC to stop anyone tampering with it.
Now here's the twist. A repeat visitor who already passed the check will be able to present the (HMAC signed) fingerprint that you gave him previously in the form of session data and he will now get the full content immediately, no delay this time
This way those bots that keep changing the indicators they present will keep hitting delays, repeatedly. The only way the bot has to not get delayed is to behave as humans do, if while trying to behave as human the bot accesses too much content it will get flagged, the bot cannot avoid getting flagged. If the bot keeps changing indicators in order to avoid getting flagged then it gets delayed every time, you give the bot no option to avoid getting weeded out.
Now the problem is that while this is quite feasible on new projects there's no easy way to do it if you use Magento, Wordpress or any of the classic publishing systems, you will have to develop a custom solution to implement this technique. This has to be done at application level.
I have been giving this some though for some time now and it should work well, it would not be too hard to do this on a new project that starts from scratch.
The classic publishing systems are totally inadequate in an era dominated by AI bots, if you insist in using them you will find yourself easy prey.