New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How do companies manage Multiple VPS locations
EfficientHosting
Member
in General
How do companies manage multiple locations like 30 to 40 country geos. DO they buy physical hardware in all these places or its just physical servers in one place and then route traffic via IP transit or IPs. As i see host companies with servers with exact same specs in more than 40 countries.
I am confused by this, how do they manage? I was thinking of getting into VPS selling business but this seems to be expensive beyond i can think of.
Comments
It really depends. In most cases, the infrastructure is either owned hardware or rented hardware. But keep in mind; companies that have 30 or 40 locations didn’t build that overnight. It takes time, a lot of investment, and careful planning to get to that point.
If you’re thinking about getting into selling VPSs, start small. Rent a server, split it up into smaller instances, and start renting those out. This way, you’ll get hands-on experience and figure out what works (and what doesn’t) without taking huge financial risks right away.
If your budget allows it, you can eventually move toward owning your own physical hardware. But even then, start small. Pick one location, ideally somewhere close to you or somewhere you see potential, whether that’s due to low latency or competitive pricing or whatever your vision might be.
Good luck!
By being efficient.
pun intended
Not OK at all
As a company that operates 6 locations, I can tell you, it's quite a bit of work.
Our oldest data center, Las Vegas, NV, went online over 20 years ago. Others are 10+ years old or almost 10 years old.
We've done dozens of major revisions to our infrastructure design: the servers that are used, CPUs, network design, and the software versions that run in them. These changes, each time, involving quite a lot of work and expense.
As other locations were built, based on the current design at the time, then all of those also need full overhauls and redesigns every 3 or 4 years, with the new design based on whatever current design we have at the time of the overhaul.
So at many points in time, we have all 6 locations operating on different infrastructure designs. It is not easy maintaining that.
As of this time, four data centers now all have our latest infrastructure design and two more are up for a redesign in the next 6 months. We have not added new locations until we can get all our current six on the exact same architecture.
But being around for a long time adds a major challenge: keeping up with technology.
As an example, let's discuss routers:
We started with Cisco GSR 12000 series. Then we went with Cisco VXRs. Then later on we switched to Foundry Networks MLX routers (amazing boxes). Then we upgraded those to their MLX-e advanced services models. Then, to clusters of Brocade CERs. Now we are on Junipers. And as of right now, we are now experimenting with vpp/dpdk based Linux routers running frr.
Each of these changes involved quite a lot of time and MONEY.
As for software, we started with Virtuozzo, then moved on to OnApp (worse decision ever) and in the end, we got sick and tired of dealing with the problems of software platforms designed by someone else and decided to create our own: ColossusCloud.
Within our platform, the first versions were based on the Xen hypervisor and now everything is KVM based.
But within KVM, we've been through many revisions and many Qemu versions, as well as many different revisions of openvswitch and now we are on overvswitch + dpdk on many hypervisors.
And as time goes by, we have to slowly maintain several software infrastructures at a time. We still have hypervisors running on Xen for clients that just haven't yet jump to our newer KVM based hypervisors. We have older KVM based hypervisors with clients that we can't directly migrate to newer ones and now we have hypervisors with dpdk, to which we also can't transition older KVM clients to KVM + dpdk because that would involve rebooting everyone. And that would be a nightmare for us and our clients.
Everything is a lot of work. For example, something as simple as network cards: we based most of all our hypervisors on Intel rj45 cards, then to X520s (fiber, which involved changing all our switches across all locations from rj45 to fiber) and then to 40g fiber (another major change). As time went by, we discovered that those Intels can't keep up with the modern requirements of high packet processing. Thus, in the last year or two, we've switching everything to Mellanox ConnectX 5 with hardware based flow offload and we are now experimenting with Mellanox DPUs.
And each of these new developments involve doing upgrades at 6 different locations and running many different infrastructure designs simultaneously.
And I wouldn't even want to discuss Ceph... that would be a whole other long post...
Yeah, it's not easy... and it is quite expensive...
You use data models to automate management
Companies rarely manage VPS instances manually once they grow beyond a few servers. Instead, they use control panels, configuration management, or orchestration systems such as. Infrastructure as Code (IaC)Tools like Terraform, Pulumi, or Ansible define servers and configurations in code, so deployments in multiple locations are consistent and repeatable. In short, companies manage multiple VPS locations by centralizing control while distributing operations.
So physical servers with own IPs. Needs lots of investment. Got it.
..quite a few small ones have "fake" geo locations and not a real physical data center space/infra where they claim to be.
but I appreciate @serverpoint long and detailed post above - it will probably save millions for folks who want to jump and skip some steps for building good network/DC infrastructure