All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Very disappointing limitation in Hetzner Cloud: max 100 servers per private network
I am the author of an open source tool that is the quickest and easiest way to create and manage Kubernetes clusters (k3s specifically) in Hetzner Cloud (https://github.com/vitobotta/hetzner-k3s) and today I learned, thanks to a user of my tool, that Hetzner limits to max 100 servers TOTAL in a private network. I am extremely disappointed because this makes it impossible to create large clusters with more than 100 nodes.
https://docs.hetzner.com/de/cloud/networks/overview/#limits
Did anyone here know about this? I really can't believe this. I can create a network and multiple subnets each supporting lots of servers, yet you can only actually attach max 100 servers to the whole network.
My tool uses a private network for secure communication between the nodes, and I love that in the Hetzner firewall I only need to configure a rule to allow all the traffic in the subnet.
To work around the 100 servers limit I could just use the public network interface, but that would mean having to maintain a much more complex firewall.
As an alternative, I could try and set up a virtual private network across the nodes. Is there something like ZeroTier that can be set up in a scripted/programmatic way on Linux nodes without having to deal with complex firewall management?
I am glad I discovered this today because I was going to suggest it at work to migrate some dev/test clusters from Google Cloud to Hetzner using my tool, which would result in savings of up to 90% compared to Google Cloud. We would keep GCP only for production.
I can still use Hetzner for dev/test clusters but I need to work around that limitation above since even with these clusters sometimes we need many nodes for load testing etc.
Comments
Setup your own VPN mesh network.
The only downside is prob. that it won't use dedicated links, no idea if Hetzner does though.
What kind of network? Can you give me an example / point me to a guide or something?
The problem with a mesh, I think, is that I would have to manage a firewall on each node. Is that correct?
Any VPN tutorial will do. Every client you connect to your VPN should have a private address only accessible by other clients on the VPN. I use Pritunl for this myself.
Depends if you use a firewall, Usually its jut a VPN, doesn't have a Firewall.
Traffic can flow freely.
I understand VPNs in general, but unless I use something like ZeroTier, I have to set up and maintain a firewall on each node, right? Hetzner's own firewall is very limited in the max number of rules etc, so I'd have to maintain a firewall on the nodes themselves. Is there a way to avoid that?
I don't think it would require that you configure a firewall any more or less than Hetzner's private networking would.
I would need to use a firewall on the nodes if I end up using the public interface, right? Otherwise stuff would be exposed to the Internet when it shouldn't.
Just bind internal services to the VPN issued IP, no firewall needed.
Usually you bind your services to the VPN IP, so it should not listen publicly.
Firewall is just there to make 100% sure nothing goes outside.
If I use wireguard or similar with the public interface on the nodes, I don't want to risk exposing Kubernetes components directly to the Internet, and I want to be able to restrict access to both SSH and the Kubernetes API to specific networks. At the moment since I am using Hetzner's private networks, I am using Hetzner's firewall to lock everything down apart from traffic between the nodes on the private network.
If I use something like wireguard with the public network, I would still need to maintain a firewall on the nodes to allow traffic between the nodes in the mesh, and that can get messy and slow very quickly with larger clusters.
I don't want to take any risks. I use k3s as Kubernetes distribution because it's lightweight and fast to install, and I don't want to be constantly monitoring what its devs do to ensure that no services are bound to 0.0.0.0 for whatever reason.
What if I use a wireguard gateway server (instead of a mesh setup) in HA mode like described here? https://noteblok.net/2022/07/28/a-highly-available-wireguard-vpn-setup/
Would performance be absolute crap?
@Hetzner_OL
Use 2 HA gateways, for the rest, drop all traffic going from your wg interface to your primary network interface.
And on the gateways, you allow certain ports, done.
My understanding from the article is that with this setup connections between peers are not dropped, but wouldn't performance be total shit if all traffic of many nodes went through a single server?
I would love to hear from them. This limitation came like a very cold shower today. I am glad I didn't do a migration yet!!
Internally, I would use a mesh network, so the traffic ideally uses the shortest route.
Externally, you have your 2, 3 or whatever gateways, which is easier to manage too.
If I go the mesh route, is there any way to prevent linux services from binding to 0.0.0.0?
Not that I know.
Any port higher than 1024 does not require higher permissions to bind.
I would not say its impossible, but likely it is.
https://docs.hetzner.com/cloud/networks/overview/
"Up to 100 server per Network"
https://docs.hetzner.com/cloud/networks/faq
"You can attach up to 100 servers to a Network.
https://docs.hetzner.com/cloud/networks/getting-started/creating-a-network/
"Choose up to 100 servers"
Friendly reminder to always check docs before you want to do anything critical, especially if you want to deploy company infra on it.
There are more limitations btw so visit these links so you won't be surprised again next time
What can I say, I have used Hetzner for many years and even built a tool on top of it, and I came across this limitation today for the first time. Do you always ready ALL the available documentation BEFORE using any service?
How is that different though? It's not like your servers with Hetzner's private network don't have public interfaces. You're doing something to disable traffic over the public IP already, same work only mildly different at most.
The difference is that if I use their private network then I can use their firewall with just 5-6 rules to manage and protect the whole cluster
Why can't you use their firewall in the other scenario?
Seems like you can have more than one network, can't you just split your things into more network(s)/groups? Does every server really need to talk to every server?
There is a limit to the number of rules etc
Because things can get very complicated with Kubernetes in this scenario
I was just reading about Headscale. Does anyone have experience with it?
Right but you don't have to firewall off services on the VPN IP any more than you have to firewall off services on the private network IP. Those services are equally hidden from public traffic in both cases.
If you're afraid of things accidentally binding to 0.0.0.0 then you have to firewall them off on the public interface the same way in both cases, or they're equally publicly accessible in either case.
Basically the same thing we were talking about, a VPN.