All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hetzner DDoS issues in Production
Considering Hetzner dedicated, for a non-gaming B2B solution. Need perhaps 100+ production servers. I have a major concern about Hetzner DDoS mitigation, which uses Arbor Networks hardware. I believe it kicks in after 2-5 minutes, which is fine. I can handle application layer attacks via nginx and some custom method. My problem is: DDoS seems to mess up the SSL handshake.
I have not experience this myself, but I have found three threads talking about this. One person said that moving to OVH solved this problem. But I do not like OVH.
- https://www.lowendtalk.com/discussion/162831/cloudflare-error-525-ssl-handshake-failed-on-hetzner-server
- https://www.webhostingtalk.com/showthread.php?t=1741692
- https://community.cloudflare.com/t/random-http-525-ssl-handshake-failed-errors-that-go-away-after-20-minutes/184232/5
Does anybody have first-hand knowledge of this? Was there a solution. I do not use a third-party service like Cloudflare, nor do I want to.
Also, are there other major problems with using Hetzner for a production solution?
Thanks!
Comments
Hetzner's DDoS protection is adequate but by no means perfect.
I've not experienced these problems myself - but I have heard of a few people who have had issues with it.
If you are in such a large scale in production - it would be better to look for someone who would take the time to make sure their solution works for you.
I've got servers at both OVH and Hetzner.
DDOS protection at hetzner is not great. If very easy to even down their 10Gbit servers with a high volume attack. However the price/performance is great and their support is fast.
OVH has way better protection but their support is the worst I have ever seen. Both have their compromises.
If you have no issues with using your phone to call OVH for support-related matters, then you're good to go.
@LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?
You aren't without direct permission from everyone involved in that chain.
Someone's going to burn the bridge with Hetzner.
Let him do it.
@jordynegen11 I truly loathe OVH. I wanted to like it. Even their admin is terriblly slow. It feels like 1999 dialup. Does OVH have an other show-stopper problems, besides terrible support and slow admin? Do you see down-times? One potential idea is to go with OVH with multiple redundant servers. I really wanted to go with Hetzner, becauses it's so slick. But the SSL DDoS issue is truly a show-stopper.
Thanks for the heads up. Do I contact support and ask them: "We need 100 servers, can we run DDoS tests?". Does that sound right, or is it laughable?
Laughable.
Yes, OVH is literally on fire.
https://yoursunny.com/t/2021/OVH-halt-and-catch-fire/
Yeah, I know the fire issue. But to their defense, that was truly a one-off thing. And, I can plan for this with servers spread out in 2 of their data centers.
Why not tell them your use case and concerns?
"Hi there. I'd like to buy some stuff from you, but first let me shit in a bag on your doorstep and set it on fire. Is that OK?"
Why not actually call them and get a real live account manager. Even at the cheapest if you are looking for Dedis thats at least 4500 a month for the 100 servers. Plus who knows you might even get a volume discount and other perks - assuming this is all legit and not theoretical
They even have a custom solutions page these days. https://www.hetzner.com/custom-solutions
What you mean is probably that the first connection is rejected. That happens due to TCP syn auth to distinguish between legitimate and spoofed syn packets. A legitimate client normally attempts a reconnect, then the connection is allowed.
You could ask Hetzner if they can turn off the TCP syn protection for you. But I guess, they don't make adjustments.
Absolutely will. But it is savvy to get an independent review. They will likely play down this issue of DDoS SSL failures to get the sale. So, I don't want to contact them, until I've done probing myself.
@mustafam Or they may actually help you test it, you know do some engineering and make a client happy. A vendor is a partner- not an adversary- unless you are going to turn and burn out of there after 30 days, or not hit any agreed commitments
I'm not running game servers or any kind of UDP data stream that suffers from latency spikes. However, my usage does involve a large number of frequent and small connections (email). Hetzner DDOS protection has been more than adequate. I've had a few short outages from a DDOS but I've had them everywhere, no network excluded. You're going to have them everywhere unless you just don't piss anybody off (or end up on the other side of a dice roll). If the perfect DDOS protection existed and could be shared on a reasonable budget, the methods used to perform DDOS attacks would adjust.
It would be a red flag that you're a shitty customer.
Contact sales, explain your current issues and how you'd like to stress it to know if it works better.
Are you English as a second language and can't be bothered to write a paragraph about who you are and what you need? I think someone more responsible than you should do the talking.
Obviously, when I do inquire, I would phrase it professionally.
Congrats on having a product big enough to need 100 dedicated servers!
Shouldn't these 100 servers be divided across different data centers and providers instead of 100 servers with just Hetzner? Like 25-25-25-25 so 4 different dcs and providers.
Is it cheaper to run it on dedicated servers and worrying about faulty hardware vs distributing it across different regions on a public cloud or clouds?
Any reasons for totally avoiding Cloudflare? I know some people don't like it but they do this stuff all day.
As yoursunny mentioned, OVH just recently conducted major cloud migration which should rule it out of any production stuff.
@kassem thanks for the congrats!
We are planning to distribute in 2 Hetzner regions.
About VPS: we need a ton of bandwidth, plus we don't like noisy neighbors. We're not storing data locally, so we couldn't care about faulty drives.
About Cloudflare: can't use because of the dynamic nature of our apps. Also, adds one more moving part. But, we'll do a re-evaluation.
About OVH: it was just one data center on fire. And we were thinking of their Montreal data center, which I believe is fairly stable. But again, OVH is the the very last choice. Their support + admin are super slow.
I understand that. Support is terrible and as you can read here, the Ipv6 network of OVH is a drama.
But I never experience any downtime besides some motherboard failures. The monitor service of OVH is actually pretty good. Most of the time faulty components will be replaced within 2 hours.
You can also create a GRE or wireguard tunnel between OVH and hetzner and route DDOS protected OVH IP's to your hetzner servers
. We ectually use that for our budget game hosting. but that means more points of possible failure.
How can you say that with certainity? The other datacenters most likely dont have better protection against such incidents. Remember their "datacenter" basically was a four level shed with wooden floors and no fire prevention at all. Let alone the "datacenter" right next to that which was basically made of shipping containers. When the fire brigade arrived they could do nothing but try to let the building burn down in a controlled way.
https://www.ovh.de/images/news/rbx4/datacentre_rbx4_ensemble.jpg
If you look at Hetzner datacenters for example, if a fire occurs there it wouldnt spread to the whole datacenter quickly (as it has only one level, and fire-proof material), so the damage could be minimized by the fire brigade, as they can access the burning parts and separate them from the still intact ones.
https://www.hetzner.com/de/assets/Uploads/unternehmen/datacenterpark-fsn.jpg
You, know that's actually very good to know. I've read similar comments on HackerNews and LowEndTalk. Baically, support + admin are terrible at OVH, but uptime is rock-solid.
Good point. The thing is, it doesn't matter. We're gonna have redundant boxes in a separate data center. We'll switch to those in case of a fire in the main data center. It will act like spare tire. And, any DB data is replicated anyways via point-in-time-recovery; uploaded to durable storage: S3 or OVH object storage.
If a fire happens in 2 data centers at the same time, no problem still. With our one-click DB recovery, we'll just shift to a different data center or at this point, probably a different provider.
@mustafam The other thing is- is your credit good enough, have you been in business long enough for them to commit 5 grand a month in hardware to you - with post pay billing? or can you prepay?
@databoss
Prepay. The cost is peanuts compared to AWS. Surprisingly, it's less than an employee's salary.
It's really hard to find a provider that would cater to both hardware and specialized DDoS protection needs. Perhaps some remote protection via Path would work? They have L7/HTTP filters as well prob have some solutions for the "dynamic nature of [your] apps". Or perhaps Magic Transit by Cloudflare, but it wont be cheap.