New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Large HTTP flood
RobertClarke
Member, Host Rep
Just under 100 million "pageviews" in the last few hours according to Cloudflare. Apache is currently pushing 997 requests a second. Running Cloudflare railgun on the site, but it's just not holding up.
Any advice to keep the site up without breaking the bank is much appreciated. To confirm, this is an actual attack on the website craftstats.com, not a lage amount of legit traffic.
Comments
Someone is probably disliking your services. Have you looked at the main ones? GigeNet, black lotus and such?
@blergh_ Site doesn't make enough for that to be worth it, but it does make enough to make Cloudflare business worth while.
@Jack Yes.
@RobertClarke in ServerCrate the 4GB plan says: "1 IP ADDRESSES INCL"
@RobertClarke i have been following what seems to be a higher than normal level of recon and other bot net activity lately .. keep up to date on auto shun and others, you are not alone!
Generally you can expect Cloudflare to start detecting and mitigating HTTP floods in roughly 6-24 hours from when they start, at least in my experience. It might very well solve itself that way.
@joepie91 i like their approach too, why I will always offer to *invest in a good host, +1 for infrastructure!
What's your budget for this. I might have something for you, however it would exceed the LEB limits.
Guessing apache is overloaded because when I try to visit I get Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data, and cloudflare doesn't seem to be doing much good at this point in time
sorry about your issues.
@Alex_LiquidHost What is it?
Weird
Nah, it is not based on LBs. On the other side it is still in early beta, been writing a WAF for myself for some time now, I might get something finished the following couple of days and will hit you up if you are still under attack.
Anyway, why apache, change to nginx and see how it goes. Also tweaking the apache itself and isntalling a couple of mods can help deal with some basic http floods tht otherwise would have taken down the box.
A nginx + apache combo may hold up better against the requests and if you are using cPanel it is easy to setup.
nginx admin
mod_evasive if they're from a limited number of IP addresses.
Edit: and also iptables limit number of connections per IP
Or throw up Varnish in front of Apache instead. Less configuration.
I've got a 5 letter word for you.
NGINX
Also, look in the logs. Usually the attack has a very specific pattern that you can use to safely block the attack, such as POST to / from www.google.com (invalid and impossible request).
He's using cloudflare.
Apache+Nginx...why not just nginx?
php-fpm if you need it.
@raindog308 easy drop in replacement. Takes seconds.
You're getting flood through CloudFlare?
It may be worth trying nginx.
At the very least put Varnish in front of Apache
Unplug it.
Trying some new stuff, not saying exactly what we're trying, as we still have no idea who the attacker is (botnet), but it should be better than what's on there now.
@RobertClarke Have you tried working with the DC / CloudFlare to see if you can get the offenders' IPs? Or are the all spoofed?
As for me, is interesting, how much RAM eat apache to do 900+R/Second;
I have experience with ~ same, but my numbers was ~ same;
What i was use?
sysctl.conf - configure your tcp-window, other options for high load;
NGINX + FastCGI only, AND ONLY! + good NINGX config = make your site not dead (with same hardware what userd for apache) and under 10000 request/s; (caching help a lot);
caching save our word.
I am done.
Quickest and simplest way is to do Varnish in front, and then you can use the varnishlog to see the url patterns and begin blocking them.
+1 Varnish.
Nginx and PHP-FPM are awesome, but takes some configuration.
Varnish, on the other hand, is very easy to add in front of Apache. Takes 5-10 minutes of your time. Make sure things are hitting the cache (
varnishstatcommand) and you should be able to handle 1000 req/s.If you can recognize and collect a list of the offending IPs, then you can use curl against Cloudflare's API to block them on their end.
ddos.sh with apf will knock down floods pretty quick.
/removed
Switched to a BuyVM filtered IP, took 20gbit, IP got nullrouted. Now we're back searching for a solution.
Get a server at awknet or somewhere serious?