New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HPC minicluster on OpenVZ or KVM possible?
i need to setup a small HPC cluster (1 head + 5 compute nodes) for my office.
however i have no experience but would like to experiment on budget openvz / kvm plans on my own before commit myself to office environment.
is it possible to setup above using openvz or kvm... assume i purchase 2 vps plan (1 head and 1 compute node).
Comments
I think you need minimum 2 compute nodes for testing.
You need at least 1 master and 2 slaves, KVM will be the best option.
Use the same provider for all your VM's and the same datacenter. otherwise your going to have network issues.
Also monitor your CPU usage on all your VM's, if you exceed your CPU usage your provider might end killing your VM's or your contract.
Look for dedicated CPU usage, maybe a slice (4096 or higher) from BuyVM will work @Franciso could clear that.
Or try some @exception0x876 wishosting.com offers
If you just want to experiment, why dont you simply run few VM-s on your local PC? It will be easier, faster and free...
i prefer to a little maybe subscribe 3 months KVM plan to proper simulate actual servers environment.
Thanks for the info. i will look up if i ready need 2 compute nodes.
What are you going to be running? Which job manager are you going to run?
>
Which kind of misses the point of HPC. You just need a little more umph, so you turn it up to 11.
>
>
Both of these.
You could also repurpose any extra equipment for the processing nodes with the headnode as a VM, or buy a bunch of Scaleway baremetal nodes, whichever.
Amazon also has a Batch Processing now. (https://aws.amazon.com/batch/)
just experimental only - not going to abuse any resources. all open source (free) so slum job manager will be ideal. prob sign 3 months for 3 KVMs.
going to document down every steps and if successful will mirror to my office environment.
@flatland_spider monitoring the CPU usage doesnt miss the point of using HPC, specially if he uses a VM with a fair CPU access policy.
I'm pretty sure that any provider from here at some point if hes CPU usage is high, constant and is fair share will end up kicking him out.
Also from the very beggining he mention his interest on purchasing some VM's.
@bustersg if you have extra hardware not been used (Pentium 4 with HT at least) and a switch you can use them, some time ago I setup a Beowulf cluster with some Pentium 3 and a 10/100 network switch, search at craigslist or get some RaspeberryPi's.
>
It does. The point is to run everything as hard as possible since one machine isn't big enough or is too expensive.
>
Shared resources? Yeah, no. Not with this.
I agree this is pretty much a nightmare scenario for most hosts. Lots of CPU, lots of RAM, and/or lots of disk IO (dependent on the application) which most aren't equipped to handle.
It would be an interesting project to see how quickly someone could get banned from a variety of hosts though.
That point did not escape me.
We're on the same page about looking for a provider who offers dedicated resources, if the OP is intent on VMs.
>
That's no fun.
Slurm is a good choice as a job manager, and I'm not just saying that because I'm wearing my Slurm shirt today.
Most HPC sites are moving that direction. The SchedMD team is pretty cool and support licenses aren't crazy expensive. Support licenses aren't cheap, but compared to other products, they're a steal.
>
+1
Documentation is always good.