How to migrate servers without downtime?
I would like to understand the technological part of how to solve the following problems. I'm interested in hosting a website, no VMs / being a provider.
Say I have a production version running on a webserver on an IP (VPS/dedi doesn't matter for now). I want to deploy a new version to a different machine and test it and after tests have passed I'd like to swap production to this one. This might happen multiple times a day.
I want to change the server to a different size / provider without downtime.
How would you solve this? I'm really not at home, but I had the following ideas:
DNS is super slow for it, some ISPs cache DNS for up to 24 hours as far as I know. I cannot just change DNS records multiple times a day, I guess it'd ruin experience for users.
Use FreeBSD jails and set up IP in such a way that the "right" IP always points to the actual production jail. I have never used jails, but as far as I understand it'd be an instantaneous switch without downtime (few seconds are ok).
Use two "big" machines and a 3rd "small" one for routing. I believe this is what HA Proxy is used for, isn't it? I only need to be sure that all 3 of them are in the same datacenter, wouldn't I? So for example I could use a 5 EUR Via Nano box only for routing to the "big" servers. Would this work? If so, how much would it slow down network that everything goes through the little box? Would HA Proxy be needed or something like a firewall (pf) would be able to solve it?
In most cases, ISPs DNS servers (and the google ones too) will follow the ttl value sent by the DNS servers which serves your domain. If you can't change this value (eg. you use the dns servers provided by your registrar), switch to other DNS provider (dns.he.net or Amazon Route 53 are fine).
You can set up a reverse proxy in the old server when switching to the new one, so the users that still have cached the old IP can continue access the site.
Well one thing would be databases, where are those stored. Because you could have the same files, but your DB will be important. You could create a new DB on your new server and type in your config the changed database, back it up quickly and restore the db on the new server quickly, then save the config, all within a minute.
Cloudflare over their CDN is instant, it resolves to their IPs so changing the new IP will only change it on their end as the user still goes to their servers for the reverse proxy.
I remember Amazon Route 53 being quite fast for basic dns.
I don't want to use Cloudflare, it makes the setup super complicated.
You are right with the DB, I'd need to make it separate.
So if I go with Amazon Route53 or Rage4 and I set 1 second TTL, it'd work on a DNS level. But wouldn't that make every single request have a DNS lookup? Thus it'd actually slow down the website, wouldn't it?
Does anyone have any experience with jails (or LXC or Docker)? It'd solve this problem on a single "big" box, wouldn't it?
Minimum TTL should be 5 seconds.
You should look for something like a floating IP, use HA Proxy + heartbeat on 2 machines. Set this up in 2 geographically different (but not too far away) datacenters and use DNS to update if one of the datacenters goes down.
For database you would have to replicate the data to both datacenters.
HAproxy on the old IP for 48 hours would solve the problem for you.
Why not keep both versions on the same machine in different directories / different vhosts. When the time comes to change the software version, just point the vhost to the directory with the new version and reload apache.
if you're worried dns records do not propagate, it is always possible to have your new records in place much earlier before any migration. Do not really need HAProxy, have old and new servers/nameservers running, update domain registrar records to move over, there'll be no downtime.
@zsero: as you talk about "5 EUR Via Nano", I assume you are at Online?
If your new and old servers are at Online, you could also use a failover IP: http://documentation.online.net/en/serveur-dedie/reseau/failover-ip
Thanks. I think I definitely don't want to complicate things on a DNS level, multiple IPs or simply vhosts look like a good option.
@rds100: I guess the simplest solution would be to have both versions under the same OS in different folders, although this would require having all the deployment set up to dynamically generate folder names on deployment, but it's definitely doable. So far I believe this is the simplest setup.
@Alt: that definitely looks like a good option. So as far as I'm with a provider which supports a control center / API for swapping IP addresses I can just migrate between machines quickly?
I still don't understand what is the difference between "multiple IP addresses" other providers offer and "failover IP".
@zsero If I were you with your setup and what you are wanting I would use either Luna Node or Scaleway which both have 'Floating IP addresses' which would allow you at any time you want to, to move the ip to another instance. That way, you can move the ip back and forth as you dev and as needed. This would likely be your best option for what your wanting so you do not have downtime. The only thing I am not sure of is if you can detach the ip while the server is running, if I understood correctly you should be able to on both services I mentioned.
The difference here being you can actually move the ip between instances in their interface (or possibly with API) as opposed to a heartbeat setup where the two vms would have to be on the same vlan and switch ips back and forth between them.
Hope this helps.
I've been reading online.net's docs and they mention that a failover IP can take up to 15 minutes to change in all the switches. That seems like a lot. Is that the reality or they just say it to be on the safe side?
Reverse Proxy,,,or a lower ttl...I used cloudflare ...it literally changes dns in seconds
So for the reverse proxy, can someone explain it to me that why is it better then a simple firewall?
Anything that tunnels the TCP connection to the new server and can efficiently handle thousands of connections would be sufficient. I'm not sure why (or how) you would use a firewall to do this instead of something like haproxy though.
I'll probably go with HAProxy but I thought I might simply just use a firewall for it, as it'll be installed anyway.
I use cloudflare and set it dns ttl to 2 mins. The downtime is almost not noticable when i migrate from server to server.
If running database, i would backup and restore database on new server follow by "remote ip of new server" on config to point to new server database. the clone the web site as well to new server but run as "localhost" database.
+1 on cloudflare @camjac251
Cloudflare is just about the simplest way to accomplish what you stated you need to do. There isn't anything simpler.