New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Load Balancer
Hey guys,
I am looking for a load balancing solution for redundancy and geo load balancing which is not heavy on wallet. I have a cluster of servers which host WP sites of my clients. (All thanks to LET and providers for awesome deals!)
I am using Cloudflare Free Plan with round robin DNS setup. It works fine but there is no health check in place. I can write a script to add/remove via Cloudflare API. Is there any better option? I am also not in favor of other DNS providers who does not have firewall like Cloudflare. I have other vpses where I can setup HAProxy but I think the latency will drastically increase?
Comments
CF also provide paid plan with health checks, $5 pm, allows you few customization and probably cheapest and hassle free solution.
https://www.cloudflare.com/load-balancing/
You can use Google DNS if you don't use CF proxy features. Google DNS is pay as you go model so if you don't have so many traffic, it may cost just a few dollars. In this case, Google does geoDNS to Haproxy at each region. Downside is no health check for Haproxy vps.
If you use CF Proxy feature then CF load balancer is the best bet because it can reuse for many domains.
Don't you have to pay for each domain if you want to use CF load balancer?
You don't need health checks anymore.
If one IP is down Cloudflare will automatically route traffic to another IP in your case.
"When a request to one IP address fails, Cloudflare automatically retries the request to other IP addresses associated with the same hostname. This behavior prevents end users from experiencing downtime."
https://developers.cloudflare.com/dns/manage-dns-records/how-to/dns-load-balancing/
For geo load balancing with CF you should create CF Worker in front or use CF Load Balancer.
I tried it but in practice it doesn't work, in my case even if one of the IPs returned a 521 code it still tries to access the same IP instead of the healthy one. It doesn't automatically retires on the other IP.
they charge per set of servers. many domains can share those set
Is it the same for Traffic Steering?
Isn't it 15 USD per month with geo traffic steering?
I need the proxy feature as I use the firewall rules and they are really effective for me. If I setup HAProxy on a VPS then what sort of latency addition I'll expect? Cloudflare -> HAProxy VPS 1 / HAProxy VPS 2 -> Cluster.
This is interesting. Saw some interesting solutions based on it. I'll dig.
https://github.com/xiaoyang-sde/reflare
https://gist.github.com/n-ts/73c29cc8cc3c0860feb4f6a1cdee8f91
https://gist.github.com/geecko86/36d244cb4842276e55e0b5ffdca883aa
Sad to know. It would have been the most easiest solution to work with.
Various options. A DYI script would work well but it would have to check the availability of both servers from various locations. Etcd could be used to have the related information available for your script.
Another option would be a scriptable dns objet at Bunny.net that would make it possible to automatically switch to two different subdomains (CNAME). See https://docs.bunny.net/docs/scriptable-dns-helper-objects#monitoringstatus-object
You'd then have to indicate this subdomain to CF, and as they do CNAME flattening & result caching this wouldn't imply any performance penalty and it could be used for your root as well. See https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/
Have fun!
Edit: Route53 might be cheaper if you don't have a bunny.net grandfathered account and would have to spend 1$/m there. If memory serves me well, using only one subdomain it's around 50 cents/m @Route53. There won't be much requests as CF will sit between this subdomain and your visitors DNS queries.
@pbx I have multiple domains/sites on these clusters. Yes. Availability check through an uptime monitor and adding/removing IPs through the API might be enough.
I have checked the BunnyCDN & BunnyDNS today. I like their offerings. If they had the option to set firewall rules like Cloudflare then I would have switched the domains to BunnyDNS. May be I can replicate the rules through their scriptable DNS. Any idea what sort of latency these scriptable dns objects might add?
Your suggestion seem interesting. Can you break down the flow for me to understand better?
AWS/Google DNS will get expensive with high amount of domains.
Not much. Can't tell precisely.
You keep most of your domains DNS at CF.
One domain uses Bunny for its DNS.
You setup a subdomain there like bunny-script.mybunnydomain.net.
And your multiple domains use it, with a CNAME record like mydomain.com -> cname -> bunny-script.mybunnydomain.net
That's what yahoo and a bunch of pretty big tech company do: no need to have the "routing system" enabled for all (sub)domains when you can maintain a subdomain and CNAME to it. You can check this working typing for example "dig www.yahoo.de @8.8.8.8" in a terminal...
you can setup from Cloudflare LB to Cluster directly. It doesn’t make sense to have another load balancer for no reason.
However, if your cluster have 20 VPS and only a subset of them have a specific website, you will need multiple origin servers on CF and it may cost more. In this scenario, using CF LB + a few HA Proxy VPS to route traffic based on domain can reduce the cost with additional latency trade off
I tried a similar approach with Google GeoDNS on one domain and multiple domains on Cloudflare Free.
In your setup, Cloudflare will cache the DNS results from Bunny DNS. The problem is the cache. If you set it for too long, you will lose the benefits of health check on Bunny Scriptable DNS. Set too low then there will be many extra latencies from CF to Bunny.
In my option, it is better to pay for Cloudflare LB. It costs more but better latency and simple maintenance
Are you sure? When I contacted CF support they told me the price was per site, hope you're right btw.
I think the billing for DNS requests is per site. But the $5 subscription is for the health check which is limited per account.
Check https://support.cloudflare.com/hc/en-us/articles/115005254367-Billing-for-Cloudflare-Load-Balancing
"Note that the first 500,000 queries are based on all active Load Balancers in your account, not per site (domain), as Load Balancers can be shared across sites by configuring a CNAME record."
"The $5 subscription allows you to configure 2 origins per Cloudflare account, 5 origins per pool, 60-second health checks, and checks from one (1) region: ideal for straightforward load balancing or failover. Different pools containing the same origin IP address count as distinct origins for an account."
Thank you, so if on top of that you enable Geo Steering you would only be able to direct traffic to two different origins?
No all the servers are equaled spec and will host the same site. I am striking HAProxy or any such additional layer off the list. It might add more latency & complexity than I thought initially.
I had thought to shift the sites to BunnyDNS, use their scriptable dns, cdn and be in peace. But you cannot use their CDN if you are using their scriptable DNS.
I am no longer considering any other DNS than Cloudflare.
I think I will also need session affinity (sticky sessions) for woocommerce customers which I believe is not possible with Round Robin DNS and will need to go for Cloudflare LB?
I am also ready to make a different implementation if that reduces the complexity like master(read/write)-slave(read) since redundancy is more important than geodns. Open to suggestions. If nothing works out then will sign up for Cloudflare LB today without the geosteering addon and be in peace.
Geo Steering costs another $5. You can have 2 origins (or pools) with 5 servers on each origin. If you have so many VPS, you can put HAProxy VPS to these origins (pools) and the HAProxy takes care of the routing to web servers.
Make sure that you understand the billing of Cloudflare LB. There are also costs for traffic.
Well it depends, if you can handle a few minutes of downtime, then a 10 minute cache for example will do wonders.
Another solution is to have all records at bunny (if CF reverse proxy is not needed).
Why not? You can use a CNAME with the scriptable DNS and it could redirect to a Bunny storage zone or whatever...
I also think that it is worth trying BunnyDNS for a small site without Cloudflare or any HAProxy first. If it is not good then purchase Cloudflare Load Balancer later.
If you use BunnyDNS Load Balancing and Geo Routing is included, right?
I don't think there currently a way to do complex load balancing, but you can remove an offline server automatically.
Check https://docs.bunny.net/docs/scriptable-dns-helper-objects for what can be done currently. If you want to do complex load balancing on the cheap you could use bunny to remove offline servers and then run haproxy, it uses very little ressources and would make it possible to redirect your traffic more precisely to various backends.
Why should I need to remove offline servers? Wouldn't haproxy already know not to send traffic to a node that's offline?
Is it possible to use Bunny DNS + CF CDN without using CF's DNS?
Look like it is not possible to have session affinity with Bunny DNS.
It is a DNS anyway. So, there should be a Haproxy server for the session affinity feature
I don't know how to setup session affinity with Bunny DNS. I think the best bet is to have a main Haproxy server for each region. If it fail then BunnyDNS can route to fail over Haproxy. Then, the Haproxy can manage the session affinity for you by routing to web servers.
You can use the CNAME trick from Cloudflare to a domain on BunnyDNS. Cloudflare DNS -> Bunny DNS (can cache for a few minutes) -> Haproxy -> webservers.
However, you can use Bunny CDN + Bunny DNS and skip the Cloudflare DNS.
In case you have several haproxy servers and one is down. You'd have to remove its ip otherwise it'll stay in the "round robin" (which is likely not a problem as most browser will try the second one if one is temporarily offline).
If you go the haproxy + round robin route you don't need to do anything at the DNS level: the browser should deal with the round robin if one host is down and haproxy will manage the rest.
If you have only one haproxy then this node becomes a single point of failure. Which might not be such a big deal if it's with a solid provider and very unlikely to do down, but if you want to avoid that that's where the ability to switch to another record automatically comes handy.
No. CF CDN implies CN DNS but you can do CF DNS -> CNAME to Bunny DNS
I agree. That would be pretty solid and very efficient latency wise. Users can be routed to closest haproxy server, and eventually to another one if the local one is down and the second one can handle the extra load.