All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Poor man's HA VPS cluster
Hi!
I am assembling a Proxmox cluster for our new high performance VPS cluster.
We help eCommerce customers grow and need as much performance as possible for Magento 1x, 2x and WooCommerce shops.
I'd like to share my setup plan and if there is interest I can share my findings, benchmarks, NVMe speeds and answer any questions you may have.
I want to achieve
- Have a really fast NVMe and SSD cluster for our KVM VPS customers.
- To be able to add more nodes easily, have backup and replication. Customer will be able to choose between SSD and NVMe storage. Speed is crucial.
- In time, we want to implement Proxmox API and offer a nice dashboard to customers for basic VE operations.
I don't want
Have a $100k+ HA solution with too much complexity and expensive & complex storage (Ceph).
Idea
Since HA is simply too expensive and complex for us I'm thinking of using ZFS replication as a "poor-man's version" of HA.
All servers use Dual Xeon E5 20 core CPUs and 256 GB of 2400MHz RAM.
This cluster consists of:
- 4 Proxmox nodes to start KVMs on
- 1 Proxmox "replication" node that all 4 nodes will replicate on
- 1 Proxmox backup node with larger SATA pool(s)
The main 4 nodes will have:
- Proxmox boot pool: 2 x small SSD for Proxmox
- SSD pool: 4x 2T SSD, RAID-10, going through PERC H330 (HBA mode)
- NVMe pool: 4x 2T NVMe, RAID-10 with separated device for SLOG - a very fast, small Intel Optane connected to PCI-E riser
Replication node is identical to other nodes only with larger SSD and NVMe pools so all 4 nodes can replicate to it.
After initial VM replication on the target node ZFS periodically sends delta changes to keep the replicated VM up to date.
Periodically = minimal interval 1 minute.
In case a primary node fails (upgrade, maintenance, etc) we can run the VMs on replication nodes.
I already have
I'm a Dell fan so I have already bought a couple of 10 bay R630 with 4x NVMe support.
I also already bought the following NVMe and SSD drives:
SSD pools
NVMe
PCI-e SLOG device
Dillemas
Am I doing this right?
Will this be slow (ZFS) or not work at all?
Should I throw $100k for a "proper" HA with storage I'm unable to maintain?
Will I be able to achieve good "NVMe-like" performance with ZFS? Should I go with LVM instead? But how will I replicate then?
You think having SSD and NVMe pools in the same machine is bad idea?
Proxmox... would you choose something else for the job? Like SolusVM?
etc.
- What to do you guys think?24 votes
- looks good, wouldn't change a thing33.33%
- id rather die20.83%
- rofl u crazy16.67%
- can't be done under $30k29.17%
Comments
I would divide the hosts into those with SSD and those with NVMe.
I recommend products from VMware. But this is a different price range.
You only write about servers and don't forget that if you want to have full HA, you also need to remember about some network devices and firewalls (I recommend Fortinet devices).
Congratulations on your first post
Instead of 4 server cluster for HA. I would recommend you to go with 3 server instead
I run my entire hosting on ZFS storage replication setup for close than year it works ....
It will be fast replication do not create that much load and drives will have native speed most of the time (faster than any networked storage)
I voted for looks good, wouldn't change a thing BUT
I would not dare to do wide mix of drives (unbalanced pools ?)
Reserve may eventually need to run so multi cores are important even it is a bit old Intel and You can make it ssd cached + HDD (log + Cache + Special ) to save on that
I use Micron Enterprise SSD where we own hardware price / tbw is good ....speed is acceptable when you add more drives ...
Read the manual carefully !
follow the KISS principle ! less complication less hassle
You need to have shared storage in HA setup. ZFS does not do this, only CEPH or NFS. RTFM before you buy hardware. And poor man HA starts from 3 servers, 4 is a bit cocky if you worry about the budget.
He will do alternative of HA, 1 minute or 5 m storage replication with RPO 1 - 5 minute
and Manual boot on the reserve server
I think what you propose will work, as long as your customers are aware of the limitations. If they are running a database with important transactions they may not want them lost/scrambled by minutes-old inconsistent backups. But if they are aware they can do binary logging to another node, etc.
Alternately look into DRBD to keep (almost) native speeds and have consistent data.
If some clients need lower RPO than 1 minute and You can add 2 VMs on different nodes with Gluster FS and attache that storage to Proxmox... (but the speed difference vs native NVMe will noticeable )
Proxmox can replicate VM disks by itself though, to make it able to boot back up off this "backup" node.
Zsync? You have better luck with ceph. And better performance. Because async is not the way.
You shoud consider fixing code to have best performance and not throwing hw to "cover" modules loading magento collection for just a slider module...
(I've been a speeker twice in meetmagento on performance issues).
Also..ever heard of public cloud? Basically each component on yout phisical cluster could fail (ram, nvme, raid controller, ethernet cables, bios, power, cpu, etc..) and without reinventing the weel , public cloud is the defacto.
Ok, small/medium budget? Any cloud hosting (vultr, hetzner, linode, digitalocean) and just use aws Cloudendure disester rexovery agent. With less then 30/40$/month you have REAL disaster recovery..
Anyway, success
Should just about be able to have 100 products on that setup, with bloatware Magento! Crappiest bit of e-commerce software ever written.