Building a New High Performance Virtualisation and Storage Cluster
In 2022, at least in Hong Kong, I'm looking to dramatically reduce the total footprint of our servers, increase density, performance and management.
Our current infrastructure in HK has been built out over more than a decade. Many of our racks are still only connected to the 'core' over Gbit ethernet. Much of our equipment is running on Xeon E5s and DDR3.
Of course we're not going to throw these things out, but it seems like it's time for some new gear and try and replace our existing 'cloud' with the latest and greatest.
Our typical storage virtualisation nodes run Dual Xeon E5 with 128GB RAM, while our storage nodes run on single socket E5s, 32-64GB RAM, 8x 3.5" hostwap bays (some nodes are SSD, some HDD and some hybrid), and it's all connected together over 10G.
Anyone have any experience, and opinions on more modern gear? I'm especially interested in 100Gbit networking. I'm trying to come up with a highly robust solution with few bottleneck points. 10G networking definitely felt like a bottleneck, but upping to 100G will probably reveal other bottlenecks elsewhere in the system.
The emphasis on the new system is density, but not at the cost of complexity. We're not limited by space, and running a CEPH/Virtuozzo Storage like storage solution added considerable complexity, and seemingly also at a performance cost as well. Our current solution is to run numerous standalone storage nodes in RAID 10, with backup nodes and scheduled backups. At least for us, this turned out to be more reliable, faster, and easier (simpler) to manage. The cost of course is a great deal of disk waste, and general excess redundancy. That said, I'm still willing to use such a model as the reduced labour is probably worth the initial up front cost. As such, I'm expecting a pretty high upfront cost to build the first initial system.
Anyone with real world experience that can share some advice on building such a system?