All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HAProxy - Is this possible?
Hi there.
I am going to be hosting a large file (legal) for users to download. In order to help cope with the demand, I will have several VPS's serving the file, The goal here is to have a type of 'load-balancer' (HAProxy?) sitting in front of the other VPS's so traffic is distributed between all servers. I currently am running HAProxy on a small vps (www-lb) with 3 servers in my webcluster (srv-01, srv-02, srv-03). I have that working, but I was wanting to know if it was possible for www-lb to sent the client directly to a server in the webcluster (so not all traffic is going from serv0x -> www-lb -> client) as I would reach my bandwidth limit on www-lb very quickly
If that doesn't make sense, please let me know and I will try to clarify it a bit more
Thanks
Comments
I believe this is possible - I don't know haproxy but this seems useful: http://serverfault.com/questions/181978/how-to-rewrite-a-redirected-url-with-haproxy
However, if all you want to do is a redirect, why not use round-robin DNS or GeoDNS instead and cut out the cost of the haproxy instance altogether? Rage4, Geoscaling or the like allow you to do a simple round-robin or nearest-first geolocation.
I was going to use haproxy as a easy way to add/remove servers and also to make sure that the content is distributed between all servers.
I might just scrap the haproxy idea and research into round-robin dns, thanks for the suggestion
Edit: Doing round-robin via CloudFlare
rr or geo dns is easier but haproxy will do it better. I can help with your config.
Thanks @AnthonySmith- I have sent you a PM.
Why not use varnish with multiple backends (enable streaming). You can add/remove backends as you go.
What you are looking for is called Out-Of-Band load balancing. The LB receives the request, passes it to the server with the lowest connection count, and the server responds directly to the client. We use Webmux Load Balancers to accomplish this, I am not able to find the capability to perform this task with HAProxy. All sources say to increase HAProxy bandwidth, as it handles all connections. Theoretically, you would be capped at 1Gbit if you had HAProxy on a gbit port, and it would be a Single Point of Failure, without geographical distribution.
@MarkTurner - I don't think varnish would be worth it as they will be downloading a .zip/.rar file, it won't be serving images or html pages ect.
@cncking2000 - Thanks for the information, from my understanding, Webmux is hardware and not software? I currently have it working with the 'redir' argument within HAProxy
@CoJustin, Webmux is a hardware load balancer, although, in taking one apart, its based on CentOS. I never tried 'redir', although functional for you, I'd think it to not be ideal for pointing the client to the closest server as was mentioned with the Geo IP comments made by @ihatetonyy Sometimes, this works out well however, as a location with decent international connectivity and multiple servers can push 10+gbit through haproxy.
@CoJustin - I wasn't suggesting to cache but to use as a 'director' to check servers, handle failover and load balance
@cncking2000 - I'll do some research today into GeoIP, chances are I'll end up implementing it.
@MarkTurner - Ah, gotcha!
As for dealing with bandwidth usage, is there some type of program/cronjob that I can use/create to monitor the bandwidth usage, once it's reached xxx GB - 'gracefully' shutdown lighttpd (letting current downloads/connections to finish but not allow any new connections)
What about serving the files from Amazon S3? You would need to use their calculator to estimate your bandwidth costs, but you could then scale your bandwidth requirements without much thought.
@CoJustin, Rage4 + geographically separated VPS (assuming this is for a global community).
It can work very, very well this way.
In addition, the newer web browsers apparently have "smarts" in them too and will perform "failover" across the various answer records provided by the DNS.
Bottom line is it becomes highly unlikely that any user can't reach a server and they have a good end user experience. (Assuming the VPS's can keep up with the loads).
I think you'd end up spending far more.
Yea, I know this is LowEnd.. but it's a solution that is commonly used for this scenario.