Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Is there a way to fail over KS1
New on LowEndTalk? Please Register and read our Community Rules.

Is there a way to fail over KS1

Two KS1 in two different dc. phpbb plus mysql on each KS1. sync with each other. Failover for one website.
Any solution to this?

Comments

  • NeoonNeoon Member
    edited June 15

    The first challenge, would be getting a KS1 outside GRA.
    13 orders, gave me 4 servers outside GRA.

    Second one is quite easy, you add 2 A records to your domain, if GRA burns down, because wood, every recent browser will try the second IP, will add some delay but you should be fine.

    Additionally, you can remove the first IP via monitoring.

    Thanked by 2letlover szymonp
  • _cece_cece Member

    reverse proxy (caddy) would be my choice

    Thanked by 1letlover
  • AXYZEAXYZE Member
    edited June 15

    Cloudflare + add both IP as "A" records

    That will rotate servers to clients.

    Now to get automatic failover you need to do healthchecks and then remove "A" record of non-working server. Cloudflare will point all traffic to working server then.

    You can either pay for CF Load Balancer to do it or program it yourself (health check + remove IP via Cloudflare API if its not healthy)

    Another option is using Cloudflare Workers as Load Balancer, see "Load Balancer with Fallback". It will be very cheap or free.
    https://blog.cloudflare.com/update-response-headers-on-cloudflare-workers/

    Thanked by 2letlover szymonp
  • letloverletlover Member

    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

  • AXYZEAXYZE Member

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    Thanked by 1letlover
  • _cece_cece Member
    edited June 15

    well for the reverse proxy you need another machine (cheap vps for a euro per month). Caddy will do the health checks for you and also issue the ssl certificate. The thing with the a records is that getting certificates with acme could be a little bit tricky, because its a round robin and you never know which machine is answering the challenge. So cloudflare with enabled proxy could be an option too. You can monitor then both ips and add or delete as you want to via their api, and you do not need to worry about the certificates. But I guess the caddy way would be the easiest.

    Thanked by 1letlover
  • ralfralf Member

    @_cece said:
    well for the reverse proxy you need another machine (cheap vps for a euro per month).

    That will probably be more powerful than the KS-1 it's proxying!

    Thanked by 1Erisa
  • letloverletlover Member

    @AXYZE said:
    Cloudflare + add both IP as "A" records

    That will rotate servers to clients.

    Now to get automatic failover you need to do healthchecks and then remove "A" record of non-working server. Cloudflare will point all traffic to working server then.

    You can either pay for CF Load Balancer to do it or program it yourself (health check + remove IP via Cloudflare API if its not healthy)

    Another option is using Cloudflare Workers as Load Balancer, see "Load Balancer with Fallback". It will be very cheap or free.
    https://blog.cloudflare.com/update-response-headers-on-cloudflare-workers/

    Thanks for the detailed explanation, AXYZE.
    How to automatically synchronize both phpbb user info and mysql database, like very 10 minutes?

  • letloverletlover Member

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

  • AXYZEAXYZE Member

    @letlover said:

    @AXYZE said:
    Cloudflare + add both IP as "A" records

    That will rotate servers to clients.

    Now to get automatic failover you need to do healthchecks and then remove "A" record of non-working server. Cloudflare will point all traffic to working server then.

    You can either pay for CF Load Balancer to do it or program it yourself (health check + remove IP via Cloudflare API if its not healthy)

    Another option is using Cloudflare Workers as Load Balancer, see "Load Balancer with Fallback". It will be very cheap or free.
    https://blog.cloudflare.com/update-response-headers-on-cloudflare-workers/

    Thanks for the detailed explanation, AXYZE.
    How to automatically synchronize both phpbb user info and mysql database, like very 10 minutes?

    I'm not sure about phpBB3 so I could be wrong, but I would:

    Synchronize two disks via rclone so they have the same phpbb content (uploads etc.)

    MySQL replica
    https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql

    You can wait for someone experienced in phpBB3, maybe there is better way :)

    Thanked by 1letlover
  • AXYZEAXYZE Member

    @letlover said:

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

    You dont need additional Caddy VPS, it wont give what you need, because not only these Kimsufi need to work, now also Caddy needs to work at the same time.

    If Caddy VPS fails and 2x KS1 still work then website wont work.
    This is method for pure load balancing, completly not suitable for your case.

    This method would be nice if you would have like 2x Caddy VPS and then 4x+ KS1. Just for 2x KS1 stick to DNS-based load balancing/failover like on Cloudflare.

    Thanked by 2Erisa letlover
  • letloverletlover Member

    @AXYZE said:

    @letlover said:

    @AXYZE said:
    Cloudflare + add both IP as "A" records

    That will rotate servers to clients.

    Now to get automatic failover you need to do healthchecks and then remove "A" record of non-working server. Cloudflare will point all traffic to working server then.

    You can either pay for CF Load Balancer to do it or program it yourself (health check + remove IP via Cloudflare API if its not healthy)

    Another option is using Cloudflare Workers as Load Balancer, see "Load Balancer with Fallback". It will be very cheap or free.
    https://blog.cloudflare.com/update-response-headers-on-cloudflare-workers/

    Thanks for the detailed explanation, AXYZE.
    How to automatically synchronize both phpbb user info and mysql database, like very 10 minutes?

    I'm not sure about phpBB3 so I could be wrong, but I would:

    Synchronize two disks via rclone so they have the same phpbb content (uploads etc.)

    MySQL replica
    https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql

    You can wait for someone experienced in phpBB3, maybe there is better way :)

    Thank you, AXYZE. Very helpful already.

  • letloverletlover Member

    @AXYZE said:

    @letlover said:

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

    You dont need additional Caddy VPS, it wont give what you need, because not only these Kimsufi need to work, now also Caddy needs to work at the same time.

    If Caddy VPS fails and 2x KS1 still work then website wont work.
    This is method for pure load balancing, completly not suitable for your case.

    This method would be nice if you would have like 2x Caddy VPS and then 4x+ KS1. Just for 2x KS1 stick to DNS-based load balancing/failover like on Cloudflare.

    Probably in the future when I know how to do this approach, I will do larger failover cluster on mission critical production site as you mentioned. Very nice though!

  • letloverletlover Member

    For the vps, I guess that I need Hetzner or Terrahost etc that provide 10 gbps, so that each user may have true 100 mbps, matching KS1's 100 mbps dedi port?

  • letloverletlover Member

    @AXYZE said:

    @letlover said:

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

    You dont need additional Caddy VPS, it wont give what you need, because not only these Kimsufi need to work, now also Caddy needs to work at the same time.

    If Caddy VPS fails and 2x KS1 still work then website wont work.
    This is method for pure load balancing, completly not suitable for your case.

    This method would be nice if you would have like 2x Caddy VPS and then 4x+ KS1. Just for 2x KS1 stick to DNS-based load balancing/failover like on Cloudflare.

    So Cloudflare and 2X KS1 is the way for my current low traffic phpbb site.
    Just 2 KS1, it is actually cheaper than 1 KS3.
    Not sure about Cloudflare fee, will check.
    Thanks.

  • ErisaErisa Member

    @letlover said: Not sure about Cloudflare fee, will check.

    Free if you use the API from another machine to accomplish what you need.

    $5/mo fllat fee if you choose to use Cloudflare Load Balancing to automate it.

    More complex if you choose Workers, but free if under 10,000 requests/day and 1,000 requests/hour: https://developers.cloudflare.com/workers/platform/pricing/

    Thanked by 1letlover
  • AXYZEAXYZE Member

    @letlover said:

    @AXYZE said:

    @letlover said:

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

    You dont need additional Caddy VPS, it wont give what you need, because not only these Kimsufi need to work, now also Caddy needs to work at the same time.

    If Caddy VPS fails and 2x KS1 still work then website wont work.
    This is method for pure load balancing, completly not suitable for your case.

    This method would be nice if you would have like 2x Caddy VPS and then 4x+ KS1. Just for 2x KS1 stick to DNS-based load balancing/failover like on Cloudflare.

    So Cloudflare and 2X KS1 is the way for my current low traffic phpbb site.
    Just 2 KS1, it is actually cheaper than 1 KS3.
    Not sure about Cloudflare fee, will check.
    Thanks.

    If you dont want any additional fee :

    Free HetrixTools for server monitoring + remove non working "A" server manually when you get alert from HetrixTools.

    OR

    Make app yourself. First do health checks, then if they fail remove "A" records via Cloudflare API. You can do healthchecks on both KS1 - first checks second and vice versa. No need for additional VPS.

    Thanked by 2letlover Erisa
  • letloverletlover Member

    @AXYZE said:

    @letlover said:

    @AXYZE said:

    @letlover said:

    @AXYZE said:

    @letlover said:
    If I use caddy to failover 2 KS1, the total price will be $12/m, less than KS3 with 2tg raid 1. Failover supposes to have much higher reliability? Am I thinking right?

    Two servers with no raid will have a lot better uptime than single server with raid, because not only you are safe from disk malfunction, but also CPU, motherboard, power, random kernel crashes etc. and if you get different location then also from network issues, fiber cut etc.

    Two servers is always way to go. RAID can fail. Two different machines in different places... not so easy.

    This means that I can use 2xKS1 and 1xvps Caddy totally $13/m, almost the same as $12.99 KS3, yet provides much better redundancy?

    You dont need additional Caddy VPS, it wont give what you need, because not only these Kimsufi need to work, now also Caddy needs to work at the same time.

    If Caddy VPS fails and 2x KS1 still work then website wont work.
    This is method for pure load balancing, completly not suitable for your case.

    This method would be nice if you would have like 2x Caddy VPS and then 4x+ KS1. Just for 2x KS1 stick to DNS-based load balancing/failover like on Cloudflare.

    So Cloudflare and 2X KS1 is the way for my current low traffic phpbb site.
    Just 2 KS1, it is actually cheaper than 1 KS3.
    Not sure about Cloudflare fee, will check.
    Thanks.

    If you dont want any additional fee :

    Free HetrixTools for server monitoring + remove non working "A" server manually when you get alert from HetrixTools.

    OR

    Make app yourself. First do health checks, then if they fail remove "A" records via Cloudflare API. You can do healthchecks on both KS1 - first checks second and vice versa. No need for additional VPS.

    Probably Hetrix tool first. Later may consider automation.

  • letloverletlover Member

    @Erisa said:

    @letlover said: Not sure about Cloudflare fee, will check.

    Free if you use the API from another machine to accomplish what you need.

    $5/mo fllat fee if you choose to use Cloudflare Load Balancing to automate it.

    More complex if you choose Workers, but free if under 10,000 requests/day and 1,000 requests/hour: https://developers.cloudflare.com/workers/platform/pricing/

    $5/m then I can use all these bmds with only one 1 tb or 2 tb hdd or 1 ssd disk, well worthy it.

    Thanked by 1Erisa
  • letloverletlover Member

    Can Cloudflare handle the KS1's 100 mbps dedi port speed?

  • ErisaErisa Member

    @letlover said:
    Can Cloudflare handle the KS1's 100 mbps dedi port speed?

    Yes, easily

    Thanked by 1letlover
  • letloverletlover Member

    @Erisa said:

    @letlover said:
    Can Cloudflare handle the KS1's 100 mbps dedi port speed?

    Yes, easily

    Thank you. Seems Cloudflare with KS1 army is the way to go.

  • ErisaErisa Member

    @letlover said:

    @Erisa said:

    @letlover said: Not sure about Cloudflare fee, will check.

    Free if you use the API from another machine to accomplish what you need.

    $5/mo fllat fee if you choose to use Cloudflare Load Balancing to automate it.

    More complex if you choose Workers, but free if under 10,000 requests/day and 1,000 requests/hour: https://developers.cloudflare.com/workers/platform/pricing/

    $5/m then I can use all these bmds with only one 1 tb or 2 tb hdd or 1 ssd disk, well worthy it.

    To be clear the $5/mo is only for 2 origin servers, it scales with how many youre balancing between. The posts by @AXYZE detail cheaper/free ways to do things.

    Thanked by 1letlover
  • mgcAnamgcAna Member

    Because you are running phpBB which uses Mysql , it will not be straightforward as you may think. There could be two scenario :
    1. Load balancer
    2. Pure failover

    When you figure out your load-balancer method, you still have things to work out like,

    In both cases you have to make sure that if user is going to either server, he gets the current data. You can't go Master-Slave configuration as suggested earlier. You will need to go master-master Mysql setup. You may also go for Galera Custer .

    In load balancer mode, you will next also take care about session consistency for user. phpBB can use redis / memcached for that. Although if database session is being used then master-master replication will solve it already.

    Thanked by 2letlover Erisa
  • letloverletlover Member

    @mgcAna said:
    Because you are running phpBB which uses Mysql , it will not be straightforward as you may think. There could be two scenario :
    1. Load balancer
    2. Pure failover

    When you figure out your load-balancer method, you still have things to work out like,

    In both cases you have to make sure that if user is going to either server, he gets the current data. You can't go Master-Slave configuration as suggested earlier. You will need to go master-master Mysql setup. You may also go for Galera Custer .

    In load balancer mode, you will next also take care about session consistency for user. phpBB can use redis / memcached for that. Although if database session is being used then master-master replication will solve it already.

    You are totally right. phpbb seems not so good at clustering comparing to other application server based apps. I will try to learn the way you mentioned. Thank you very much for your experienced insight.

    Thanked by 1mgcAna
  • NeoonNeoon Member

    So, I was bored.
    Moved my https://wiki.x8e.net to my baguette cluster.

    We got:
    1x KS1 in Gravelines 1
    1x KS1 in Roubaix 4 ( shadow)
    1x Avoton in Paris

    Could also used a KS1 in RBX5, however the DC's are to close.

    In different DC's and some on Different Networks.
    2-5ms in-between them.

    Storage is replicated with gluster, connected via mesh vpn (wireguard).
    Everything from ssl to nginx configs and the wiki itself runs on gluster.

    So no local config files or local mess.
    Round Robin with the 2 IP's will do the rest, no external monitoring needed for now.

    Will this cluster survive a possible SBG in GRA? yes
    Would it survive a possible RBX? no

    It can loose 1 node, if we loose more than one, toast.
    To compensate for a possible RBX, it would be necessary to run 5 nodes.

    Sadly, I got only 4 nodes for this, I need 5.
    I could not find any cheap other Provider on a different Network with the same pricing and specs.

    Performance wise, I managed it to bring it below 1s FTTB, however its still not performant enough, will do need to do more tweaks.

    If anyone has tips, lemme know.

  • @AXYZE said:

    @letlover said:
    So Cloudflare and 2X KS1 is the way for my current low traffic phpbb site.
    Just 2 KS1, it is actually cheaper than 1 KS3.
    Not sure about Cloudflare fee, will check.
    Thanks.

    If you dont want any additional fee :
    manual DNS update
    OR
    script up own health checks and automated DNS update

    Another hacky option: have each machine run a DNS server and be authoritative for the domain (with an extra step you could delegate a sub-domain this way too) but each only returns its own address in response to requests, with a really short TTL. Normally each machine should see traffic as the name servers are used round-robin, if a machine goes offline it stops responding so the other will be used by downstream resolvers so requests get redirected automatically (once the TTL has expired).

    You would basically be (mis)using standard DNS behaviours to handle fail-over.

    Want to drop a machine from circulation? Just stop it. Remove the NS entry if you want it off permanently. Adding a new machine? Just set it up the same way, copy in the content, and add the NS entries.

    Of course this has a number of disadvantages:

    • It only automatically deals with a full node stop/fail/disconnect: if the web server goes down but the DNS daemon doesn't fail-over won't happen (you'd need an extra health check to kill the DNS server if you want to cover that case).
    • In normal operation load balancing will be arbitrary, probably not very well-balanced at all. Everyone using Google's DNS resolvers in the same region will likely go to one of the servers by default, for instance.
    • If your services store state in files/DBs/other you'll need to keep that data in sync between the machines (though you'll probably have this same problem with any other solution). Also there are no sticky sessions: a single user's session could potentially swap between machines at any time, it usually won't but you'll need to design for the possibility, so this state sharing if needed may need to be quite efficient & reliable…

    But for static (or otherwise low-state) content where fail-over is the key feature (i.e. balancing is at most a nice-to-have), it should just work.

  • mwtmwt Member

    Here's my script that updates the DNS as described:

    https://github.com/mwt/monitor-dnsupdate-cf

    It only works for Cloudflare. It does a check with curl and changes the DNS if curl times out.

  • Shot2Shot2 Member

    @MeAtExampleDotCom said:
    Another hacky option: have each machine run a DNS server and be authoritative for the domain (with an extra step you could delegate a sub-domain this way too) but each only returns its own address in response to requests, with a really short TTL. Normally each machine should see traffic as the name servers are used round-robin, if a machine goes offline it stops responding so the other will be used by downstream resolvers so requests get redirected automatically (once the TTL has expired).

    As explained here:
    https://blog.l0cal.com/2015/04/30/using-dns-as-a-cheap-failover-and-load-balancer/

    For basic needs, it works. Throw extra features in the equation (e.g. SSL, or DNSSEC with live signing) and it becomes (w)hack-ish.

Sign In or Register to comment.