All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Switch between main and back-up server
ITChristian
Member
I manage some websites for a non-profit. The main websites are built using the Apache web server and PHP (with no database). We also have some features on the websites that use FFmpeg, C++ binaries, or Node.js. We want to offer high uptime and would like to have two VPSs on different providers: one VPS to serve all the websites and another as a backup. I think the content "clone" could be done using rsync, assuming I configure both servers the same way. We would like to use the backup server if the main VPS is down.
The question is: What is the easiest way to use both servers, and if the main server is down, to automatically show the contents from the secondary server based on your experience?
As a non-profit, we are trying to find an inexpensive way to do this.
Thank you!
Comments
Current specs ?
At this time, we use one VPS with the following specifications: 8GB RAM, 4 cores, 500GB SSD, 30TB bandwidth, and 1Gbps and another with similar specs.
But how do the specs affect how we can automatically switch between servers?
Thanks!
Use Cloudflare Load Balancer so it will auto switch over to the other domain/IP of your other server. It is a paid addon but they make it pretty easy to setup. Or you can setup a HAProxy server if you are willing to set it up manually.
https://developers.cloudflare.com/load-balancing/
Because maintaining 200-400GB of static assets is different than operating 200-400GB of database.
Rclone is pretty handy at keeping several nodes synced (files). While many database has their own approach.
If you don't mind serving static files on S3, @systemfreaks sold 1TB storage 1gbps unmetered for $36/year. That's more than enough to offload the bandwidth strain due to assets
Then, use small vps from e.g Gullo's @Cam , Tierhive @backtogeek, @Nube, @AliceNetworks to maintain the dns (or cloudflare) and do DR (provision new temporary server, set it up, synced it) in case one of the vps went down for hours
GCore provides health checks and auto failover on free plan too !! So for a nonprofit it would be better.
Thank you for mentioning
@ITChristian
The offer is still active in case you are interested you can order from here
https://manager.systemfreaks.com/order/config/index/Object-Storage-3-TB-2026/?group_id=43&pricing_id=3675
If you 2 phisical server in one switch, you could try keepalived , domain config vip, simple and efficetive, otherwise you could use ddns kind, you could self do monitor script, and when main machine fail, use script switch domain's relate ip to backup server.
I wouldn't quite set it up as a failover, but instead set up haproxy on both machines such that both machines forward to the service on that machine, and failover to the over. Actually, the way I prefer to do it is not to use failover, but just give the local one a much higher weight.
At that point, either machine can be used to access the service. If they're near each other geographically, I'd just list both machines in the DNS as round-robin for automatic failover, but failing that you could set something up to have a low TTL on the DNS records, and monitor when the first machine fails and switch to the second.
Some providers provide a fail-over load-balancer (which is essentially the same as above) if you don't want to configure it, but then you're restricted to a single provider.
Personally, I'd want to use at least 2 different providers because they are less likely to have outages at the same time.
rsync is often fine for static data, especially when it changes infrequently. But you might have issues where one file (e.g. home page) is copied and points to another page or other resources that haven't yet been copied.
If that's an issue, you maybe want to look at something like the xfs filesystem (which I've no experience of) as you can send snapshots to the other machine, and then everything should be replaced at once after it's all copied.
Another approach is to have 2 copies of the website on the each machine, served by different copies of apache. When one copy is being updated, you take that apache offline first, do the copy, and then start it up again, and shut down the other. If you add both to haproxy, it will transparently switch between them.
You'll also want to make sure that you forward certbot requests between the instance. I have one machine that all the others forward
.well-knownrequests to and it copies the certificates around to the others.We had a discussion with Google AI and identified a free solution on Cloudflare:
The daily requests quota for Workers is 100,000.
Our use case was geo-forwarding, where the Worker would choose an origin server near the Cloudflare edge.
I would like to thank you all, for your help and suggestions. For the moment, we will try the DNS Fail-Over method, with GCore, as someone mentioned. It seems to be the easiest method for now, so we will see how it works. I didn't hear about GCore before and their free DNS hosting (I used Hurricane Electric DNS hosting in the past, which offers a really nice free DNS hosting, but they don't offer DNS fail-over).
Thanks again!
A solution I use myself (because I selfhost my DNS) is that with PowerDNS (and maybe others) you can create a LUA record that checks regularly if port 80 or 443 is responding; if not, it doesn't return the usual contents but an alternate one. Has a few advantages above using RR.
Make both master and slave your only nameservers.
If dual master is okay, then each one sends only its own IP in A records (and AAAA records, hopefully.)
If you need a single master, the slave sends the IP of the master unless it can't reach the master.
The risk is if connectivity is severed between master and slave, and then you can have a runaway dual master scenario. You could also deny writes on the slave all of the time, so nothing can be written to the slave if the master is down.
Definitely low TTL in this case. This type of DNS failover works very well in practice.
I'd argue that both nameservers should send both A records. In most DNS servers, the order is randomised on replies, and client will typically connect to the first and then the second.
If you just send a single A record, then if the webserver is down but DNS isn't then clients can't attempt to access the backup server. If you send both, they would typically try the second IP address in under a second if the first one won't connect.
If the servers are relatively close geographically, it probably doesn't really matter too much anyway.
I think that @slowservers solution is elegant and not too hard to implement. In second place I see the "classical" NS IP switching (but as was said, low TTL is important!).
Btw I'm somewhat surprised that AFAIK ther's still no open source name server with hooks that make it possible to replace a non-responsive host A by a responsive host B (and of course back one A is up and responsive again).
Why would the webserver be down and not DNS? It's certianly possible, but something that should be monitored for. You could even write something to kill DNS if the webserver dies for some reason.
One of the beauties about this system is that you can kill DNS, wait a minute, upgrade everything else, and have it come back without managing state between two nodes.
I wish HTTP/HTTPS was done like MX records where you can set priority. I've found it's pretty random which one the client will connect to, and if one is up and one is down, the experience is terrible for users. I haven't tested it in a while, but it just seems like it's never worked well and timeouts were set too high.
I'm not aware of one, but it's quite possible to write a Perl script to update a zonefile for you. I think Perl has modules specifically around zonefile management that makes it easier.
If you have access to BGP,
setup ECMP based failover https://docs.vultr.com/high-availability-on-vultr-with-floating-ip-and-bgp
It works great, if you want a much more simple approach I would use Cloudflare load balancer with healthchecks!
Nice one! I used to do it kinda like this on a small setup, its very low-end and does the trick very well for small projects as long as you use a low TTL.
I would like to thank you all, for your help!
We are now using lsyncd for syncing files. We are not using databases, so it was easy to sync the files.
For the switch between servers we solved it by using the dns.he.net free service, with the A records of the domains being setted as dynamic dns. We are using a third very small server for monitoring. If the main server is down, the script makes a request to the dns.he.net API to update the IP of the dynamic dns, and then to set it back to the main server, when it is running again.
Gcore is also a good service, with free fail-over. Both solutions come with a 5 minutes update interval.
Thanks again!