Can't I use OVH for reverse proxy and caching server?

tommmy

Hi.

I am using my OVH VPS as reverse proxy to my main server and for caching. I only use nginx on that server and nothing else.
I started using it a few days ago.

Today I received an email, and my server has been terminated.

Dear Customer,

Abnormal activity has been detected on your VPS vps-xxxxxxxx.vps.ovh.net.

As this constitutes a breach of contract, your virtual server vps-xxxxxxxx.vps.ovh.net
has been blocked.

You will find the logs brought up by our system below, which led to this alert.

- START OF ADDITIONAL INFORMATION -

Attack detail : 13Kpps/10Mbps
dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason
2024.03.22 12:46:15 CET 54.xx.xx.xx:35200 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:41998 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:54898 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:50360 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:50022 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:35644 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:59338 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:56910 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:52004 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:58084 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:34990 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:58666 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:49376 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:57188 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:58524 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:40304 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:58810 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:36642 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:46188 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN
2024.03.22 12:46:15 CET 54.xx.xx.xx:46936 19x.xx.xx.xx:80 TCP SYN 60 ATTACK:TCP_SYN

- END OF ADDITIONAL INFORMATION -

OVH Customer Support.
Thank you for choosing OVHcloud Asia!

The destination IP is my main server.
Am I using the server wrongly? I opened a ticket just now and still waiting for a reply. Posting here to get some thought from you guys.

I have been using the same method with BuyVM for so long without any issues. I switched to OVH just for fun. Did I breached the contract on OVH with my usage?

Thank you.

JamesF

might be worth tagging @OVH_APAC for some clarity?

Did you get any warnings? or was it just terminated?

momkin

I've tried it before , it doesn't seems to work because ovh anti ddos thinks you are attaching your source server.

tmntwitw

From the email shown, it mentions your VPS has been "blocked". Maybe your VPS is suspended only and not terminated?

It could be possible that the very large amount of simultaneous outgoing port 80 connections/traffic in a short burst (all of which is going the same destination IP) might have triggered some alerts at ovh side. Maybe passing the reverse proxy traffic (ovh VPS -> your server) over wireguard tunnel might be a better option instead?

tommmy

@JamesF said:
might be worth tagging @OVH_APAC for some clarity?

CS9460037 is the ticket number if @OVH_APAC wanted to check.

Did you get any warnings? or was it just terminated?

That is the first and last email I got regarding the issue. The VPS is not not accessible from the dashboard.

tommmy

@tmntwitw said:
From the email shown, it mentions your VPS has been "blocked". Maybe your VPS is suspended only and not terminated?

>

I am not sure. When I tried to access from dashboard, it says service expired.

It could be possible that the very large amount of simultaneous outgoing port 80 connections/traffic in a short burst (all of which is going the same destination IP) might have triggered some alerts at ovh side.

I am proxying and caching my Wordpress server which has quite large traffic. Maybe thats why.

Maybe passing the reverse proxy traffic (ovh VPS -> your server) over wireguard tunnel might be a better option instead?

>

I have been using just nginx proxy for years and it has been very simple and working fine. But I am opened to any better way, so I'll take a look. Thanks😁

tommmy

@momkin said:
I've tried it before , it doesn't seems to work because ovh anti ddos thinks you are attaching your source server.

It was working for me until the server got blocked.

woteti

ddos prot is a double edge sword. good luck resolving that issue.

Neoon

@tmntwitw said:
Maybe passing the reverse proxy traffic (ovh VPS -> your server) over wireguard tunnel might be a better option instead?

UDP is much more restricted with OVH bandwidth wise.
Under attack you can end up at 50Mbit, prob. not wise to run it over a VPN.

rm_

@momkin said: ovh anti ddos thinks you are attaching your source server.

a) enable Keep-Alive connections to the source server on your reverse proxy, so that it's not a million new TCP connections on each request, but like 8 or 16 of those are established and everything goes through those; nginx supports that; better for performance as well;

b) set up a VPN, such as WireGuard, between the main server and the source server, and do all proxying over a VPN; better for security; combine with the keep-alive setup mentioned above, to get the performance benefit as well.

Another alternative, connect to the source server over IPv6. I don't think they have IPv6 filtering in place, or it's not as strict.

matey0

13k pps seems very high for a wordpress blog, no?
Either way, if you set up a vpn from the proxy server to your wordpress backend it would probably cause less issues.

Cybr

Try to optimize the TCP connections with Keep-Alive, or maybe try IPv6 like rm_ suggested.

Using a VPN over UDP is a no-go with OVH since as soon as you have significant traffic their Anti-DDoS will think an attack is coming from your source server and block it.

I personally gave up on proxying traffic via OVH years ago since their Anti-DDoS system just kept blocking traffic to the source server with any more than a small amount of traffic no matter what I did. I now keep either all connected servers internal to OVH or have them all on other providers, never mix OVH with others.

rm_

@Cybr said: Using a VPN over UDP is a no-go with OVH since as soon as you have significant traffic their Anti-DDoS will think an attack is coming from your source server and block it.

I think that might have been solved, I was not able to trigger that as of recently, running iperf3 over WG.

totally_not_banned

@Neoon said:

@tmntwitw said:
Maybe passing the reverse proxy traffic (ovh VPS -> your server) over wireguard tunnel might be a better option instead?

UDP is much more restricted with OVH bandwidth wise.
Under attack you can end up at 50Mbit, prob. not wise to run it over a VPN.

If UDP is problematic he could as well run OpenVPN over TCP, which of course comes with all the downsides of using TCP for tunneling but depending on how problematic UDP is in this scenario it might still be an option to investigate.

danblaze

Are you considering using http/2 as a return source? This would enable multiplexing on tcp, which is helpful. Also, I think you could get a few more low config ovh vps so that once the traffic is spread out, it might not trigger the threshold for ddos protection.

On the other hand, I think you'd be wise to host your static assets on an S3 service, such as OVH's own or R2, if you're willing to do so - a good portion of the requests will be distributed directly through S3, which can be a good way to reduce the stress on your site.

tommmy

@rm_ said:
a) enable Keep-Alive connections to the source server on your reverse proxy, so that it's not a million new TCP connections on each request, but like 8 or 16 of those are established and everything goes through those; nginx supports that; better for performance as well;

Thanks a lot. This brought down 2k+ connections to just a hundred. Blog is working fine too.

b) set up a VPN, such as WireGuard, between the main server and the source server, and do all proxying over a VPN; better for security; combine with the keep-alive setup mentioned above, to get the performance benefit as well.

I think this is beyond my skill. It sounds interesting though, so I will look into it.

Another alternative, connect to the source server over IPv6. I don't think they have IPv6 filtering in place, or it's not as strict.

Thank you. I'll keep that in mind.

tommmy

@matey0 said:
13k pps seems very high for a wordpress blog, no?

I am not sure. I am not good with numbers, but it is one of my main source of income.

tommmy

@danblaze said:
Are you considering using http/2 as a return source? This would enable multiplexing on tcp, which is helpful.

You mean HTTP/2 between the proxy and the backend? I am not sure about that. But quick Google said this.

Also, I think you could get a few more low config ovh vps so that once the traffic is spread out, it might not trigger the threshold for ddos protection.

That sounds great too. I'll decide after OVH team replied to me. I have implemented KeepAlive as @rm_ suggested and stuff looks great now. But not sure how would it played out on OVH server since they haven't replied me yet and the server still locked.

On the other hand, I think you'd be wise to host your static assets on an S3 service, such as OVH's own or R2, if you're willing to do so - a good portion of the requests will be distributed directly through S3, which can be a good way to reduce the stress on your site.

Thank you. I want to avoid additional cost as I am not familiar with S3, But I'll search around.

ehhthing

I think this might be due to repeated failed connections between your reverse proxy and your backend. It's detecting it as a TCP SYN flood.

OVH's firewall is very annoying sometimes unfortunately. I had to stop using my Wireguard server on OVH because it kept detecting it as DDoS.

tommmy

@ehhthing said:
I think this might be due to repeated failed connections between your reverse proxy and your backend. It's detecting it as a TCP SYN flood.

I didn't see any logs regarding any failure. nginx usually tell in log if upstream server failed to connect.

OVH's firewall is very annoying sometimes unfortunately. I had to stop using my Wireguard server on OVH because it kept detecting it as DDoS.

Oh I see. Thank you for sharing.

NanoG6

@tommmy said: I think this is beyond my skill. It sounds interesting though, so I will look into it.

Easy if using Tailscale. Go try it.