All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
OVH DDoS Mitigation (522 Cloudflare)
Hey there!
I have a ovh vps and I'm using cloudflare as reverse proxy.
Whenever I get at least incoming 1000pps due to L7 (D)DoS attacks or let's say on high user traffic on my website, ovh will turn on ddos mitigation and I receive an email.
While ovh ddos mitigation is active, cloudflare ip ranges will just be blocked and all requests to ovh backend will be automatically discarded which ends in a 522 timeout error.
The problem is that I really don't know how to solve this issue. I have googled a lot and I found out that ovh has a firewall where you can whitelist up to 20 entries. But anyway I don't think this will help to whitelist cloudflare because cf has even more ip's and someone in the ovh forum created a discussion that this method didn't work for him anyway.
Does anyone know what you can do in this situation? Do I need a dedicated ovh server or just other settings?
Comments
Don't think you can make CF work with OVH AntiDDoS.
Even if you whitelist IP's they may get affected by the mitigation.
And given the amount if IP's CF has, no way.
Disable CF and let OVH do the work, mitigate the rest on the VPS.
Read this https://lowendtalk.com/discussion/comment/3440092/#Comment_3440092
You should have that "CF-Connecting-IP" header fixed to get the real IPs!
Good to know but does OVH really mitigate high amount of L7 ddos? As far as I know there is no challenge based authentication like js challenge or button / captcha. As far as I could see OVH has inbuild rate-limiting but on huge botnets won't help enough I guess.
Like every common hoster has a ddos protection. I also asked Hetzner and they said on ddos cloudflare ip's can be blocked also. What kind of hoster should you take then?
Thanks for the link. The problem I'm facing is that I can't adjust the ovh anti ddos protection. If I had a self-configured nginx reverse proxy it would help of course but with ovh I don't know.
How? if you are using TLS, then OVH can't decypher the packages.
Anything Plain should be possible to be DPI by OVH, but no idea if they do.
CF can because they break the end to end Encryption, which is also a concern.
However, if OVH keeps the flood outside, all you need to do, is reinforce your application on the weak spots.
But, I guess would be easier to find a different provider who can whitelist CF.
Also, from the looks of it, seems CF fails to mitigate the DDoS, since its hitting the OVH VAC and it drops CF traffic.
Perhaps try using Cloudflare Tunnel. It differs in that you establish a long-lived connection to Cloudflare instead of Cloudflare connecting to you. OVH might be treating them differently, I'm just guessing.
Hey @AliveSurvive,
That's really weird behavior, I've been using OVH Services for some years, and never had false positives between their mitigation and Cloudflare.
Even because of OVH whitelisting Cloudflare IPs, one of the OVH bypasses that were developed in 2020 was spoofing Cloudflare IPs in order to bypass OVH Mitigation (It was patched some months after).
Either way, I'll send you a PM so I can help you verify what is going on!
Best Regards!
If CF is really leaking the DDoS downstream, then I guess it does not matter much, as soon the VAC sees the amount of packages, the entire tunnel will likely be in trouble.
But of course, its worth a try.
Maybe its even a race condition, CF takes longer to detect the attack, meanwhile VAC is already mitigating it and its to late.
To be honest I’ve never found Cloudflare to do much on its own in terms of mitigation
I start receiving 2 million requests out of nowhere within like 10 minutes and even with under attack mode using the JS challenge on Cloudflare was doing nothing. Only solution was making a firewall rule to show a “managed challenge”. In any case I think I was getting some 522 errors during the attack but mostly my VPS was on 1200 load average so obviously the Ovh mitigation was not doing anything against Cloudflare. For reference the host system was a VPS in OVH DE from @Abd
Well, that's actually a good theme, since Cloudflare released the "DDoS notifications" people have been receiving emails of millions of req/s, which seems kinda weird since an attack that reaches millions of req/s is a massive DDoS Attack that can literally kill more than half of the internet (including big anycast).
The truth is that, since this 'new system' came out, Cloudflare has been claiming to mitigate attacks with millions of req/s, like the last attack which was "26 million requests per second", well as Cloudflare Javascript challenge can't really do much about Browser Emulators, or Selenium or PhantomJS I'm wondering how 'real' these statistics are. They can't stop 100r/s of Browsers Emulators, how would they mitigate 26 million requests per second on a FREE plan? I can't believe that someone launched an attack so big and full of malformed requests, it would make no sense.
Do you have exact numbers on this one?
If the mighty VAC is not getting triggered, I guess its below the thresholds.
That wont help at all in this case.
Theres no way to forward these IPs to OVH VAC.
@AliveSurvive make custom firewall rules on CF side. You have plenty of options there, just read docs and analyze common attack vectors on your server (maybe all bad traffic uses HTTP1.1?)
If you have problem with L7 and you have enough bandwidth then use nginx and then use CF-Connecting-IP as real IP, setup rate limiting zones per IP, implement custom L7 protection like "drag piece of image on image" and if someone fails to pass it multiple times just drop the connection, you can even forward it to CF via their API and this IP wont even hit your server.
Other than that check from which ASN attacks come. If its usual ColoCrossing + DigitalOcean etc. hosting companies then just make a firewall rule in CF to give them all "Legacy Captcha". If that doesnt help just block these ASNs. Just make sure you dont block residential ASNs and youre fine.
And if you have money then just pay for someone who knows how to setup CF + nginx server correctly. As long as connections arent from IoT botnet on residential connections then its fairly easy to block or make it too slow to affect your website.
Don’t have the exact numbers right now as I’m on vacation but from the firewall rule when I checked within 30 minutes there was ~3 million challenge triggers with about 80 solved if that indicates request frequency. AFAIK VAC was not triggered but I have been targeted by DDoS for a while now without fail and VAC never really did anything anyways.
Either way it was from @Abd so I wouldn’t be getting any OVH emails
"80 solved" I would understand this as 80 passed through and CF catched 3 million fake requests. That would be way to low for OVH to react.
In a different thread, regarding Hazi, he spoke about this method.
You flood your enemy with a shit ton of fake requests, so you confuse him or pull his attention away which that would be the case and a few "legit" ones make it through.
No idea what application you run, but 80+ people hammering certain parts of your appllication may involucrated your small vps.
Maybe these numbers are even incorrect, but it sounds like a known pattern.
Do you know how many actually did hit this machine? did you checked your logs?
Never checked my logs. I am assuming the 80 solved are legitimate users. After implementing the challenge, the attacks stopped after a little while as he gave up
Personally, I have set up things like this for clients who have asked for freelance support when they are using OVH directly.
I have this script that uses iptables to block all connections on web ports apart from Cloudflare based ones
https://cdn.ifast.uk/linux/CloudFlare-HTTP.sh
Then create a firewall rule on Cloudflare to captcha challenge (not JS) ASN's such as OVH, Hetzener and add countries such as Russia, China, Japan, and India as a starting base, then if the attacks are able to bypass the current rules review the locations or ASN's the attacks are coming from and add to the rule
This can all be done on the Free plan so no real extra costs apart from your server
Why you have IPs hardcoded? You can just get fresh IPs from CF so script will not get outdated and cause problems.
https://gist.github.com/Manouchehri/cdd4e56db6596e7c3c5a
Even if 100% of the requests bypassed cloudflare, it's still effective and not doing nothing because it becomes more resource-intensive and expensive for the attacker.
This was my first script but does the job, mostly cloudflare’s ranges do not change since they own a large number of ranges
I welcome feedback in terms or constructive feedback but simply fault finding as per your post isnt really needed
If I were you, I would start fresh by creating a new hourly vps at vultr or DO. Then check cloudflare logs to block non-residential Asn and suspicious countries in cloudflare.
After that, move dns record to new vps, see how things go. If the ddos stopped, then consider changing block in cloudflare to capcha or javascript to avoid blocking legistimate users. It would be better to setup nginx front-end to limit rate per Ip. You can move back to ovh after the ddos is over.
Hope this help!
Its not constructive?
Ok, lets change that.
Here's current IPv4 CF range
https://www.cloudflare.com/ips-v4
Where's 104.24.0.0/14 range in your script?
You dont have it at all.
Why you have 104.16.0.0/13 instead of 104.16.0.0/12?
Because it changed.
xD yeah, could you write that one more time now?
I ASKED you why you have done it that way, pointed you to alternative which is more robust. You call that "fault finding"? I help you fix your script? Its already outdated as we speak, you didnt notice it and deployed it in god knows how many instances.
You should thank me instead of saying that my post "isnt really needed".
Where did you get this number from? I've never needed to look into the DDoS protection in any great detail (although I have occasionally had such emails)
But 1000pps sounds incredibly low. If you consider a maximum MTU of 1500 (and you'll be below that), 1000 packets of 1.5KB would be 1.5MB/s or 15Mbps. You must definitely be able to sustain that, or you'd have noticed before now, so the metric must be something else.
Maybe 1000 unique IPs in a second is more plausible, but whatever it is I don't think it'll be packets per second.
You misread that bro.
He isn't saying that he cant sustain that, he is saying that he gets 1500pps from Cloudflare IP and OVH blocks Cloudflare IP range.
Idk when OVH VAC kicks in, but the problem here is that VAC just blocks his whole website to everyone (because everybody connect to his website via CF), not that his server isnt powerful enough. Different things.
I'm not talking about his server not being powerful enough. I just mean he can't be talking about packets per second, because he wouldn't be anywhere near the bandwidth limit which is easily obtainable on the cheapest OVH machine. He must be talking about connections or unique IPs or something else. If it's packets, 1000 must be far too low a number.
U sure?
"By default, if you go higher then xxx PPS, the OVH VAC will detect it as an attack."
https://lowendtalk.com/discussion/comment/3300818/#Comment_3300818
Like I say, I'm not sure about OVH's DDoS system as I've never paid it much attention. But the "xxx" to me suggests an unknown placeholder value.
I'm just saying that 1000 packets per second isn't a lot. In fact, it's at maximum 1.5MB per second or 12Mbps.
EDIT: that link is also talking about UDP. I guess OVH might filter UDP traffic more aggressively than TCP. And thinking about it, I've certainly heard of people talking about losing Wireguard packets when the DDoS protection kicks in, But, AFAIK CF would be using a TCP connection to forward requests to you.
Yes, UDP is generally capped, under DDoS even more heavy.
HTTP/3 uses UDP and if he has LiteSpeed he likely uses it. We dont know exact configuration here.
I also dont know how VAC exactly works, so we need to wait for OP to give more info
Thank you all for your answers.
I'm using plesk with the standard configuration (nginx + apache). In the plesk panel you can see incoming packets on the monitoring tab. It's not 100% accurate but it was always around 1000 pps (rx) when OVH sent me an email that ddos mitigation has been turned on to protect their infrastructure. That doesn't mean cloudflare ip's will be directly blocked but if the attack continues, they will.
I'm sure that ovh blocked cloudflare ip's, because as soon as the attack continues, you aren't able to connect via domain anymore (even to the plesk panel) - 522 timeout. If you enter the plesk panel directly via ip then you could connect. If you turn off cloudflare proxy mode for a domain, then you could also connect. As soon as ovh turned off ddos mitigation you could connect again via proxy mode.
I have also contacted ovh but the response time and quality of response was very disappointing. The support didn't even check my case. I only got a standard template response that ddos protection cannot be disabled and if legetime traffic has been blocked I could send some details. No word regarding cloudflare.
So even if just a single booter flooded the website with get requests, ovh would only recognize the cloudflare ip's as danger and rate-limit them.
Btw. I have a ovh cloud vps in poland warsaw.
I have been using cloudflare with OVH VPS and it seems ok…. The websites aren’t high traffic, so that maybe why I haven’t seen this yet.
Are you using the free cloudflare plan?
We are providing our protectiong for free for a limited time if you want to give it a try send me a PM