New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
RTX 4090 8x Bare Metal
Hi,
I'm renting a number of 4090 GPU BMs at the moment for a good monthly price but seeing if anyone else can offer similar pricing on a smaller deployment timescale.
The requirements are (if providing, e.g. 8x 4090s in a single bare metal server)
- At least a 64 core CPU
- At least 192 GB ram
- At least 1TB SSD (raid not needed)
- 8x 4090 GPUs
- Out of band access
- 9 Dedicated Public IPs
- Location: preferably east coast US but US in general either
I would install Proxmox and create VMs giving flexibility.
It would be rented via a business so would need invoicing etc.
Thanks
Comments
Afaik, it is not possible to mount 8x 4090 on stock mobo. I asume you will pump those poor cards to 100% so power consumption alone would be humongous.
We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.
However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.
It's mainly so I can manage the VM templates that I have everything installed on and able to get into the VMs via console. Do need Ubuntu 22.04 for example but I think you install Ubuntu 24.04 at least from the dropdown options.
Right now I'm running all VMs with 1GPU so wouldn't be the too bad if you had Ubuntu 22.04 as an option.
We can definitely provide Ubuntu 22 and do custom templates, I'll shoot you a PM! We also allow users to mount custom ISOs.
Great, thanks.
I’m a bit of a newbie when it comes to GPU hosting, but I have always read that multiple GPU’d dont mean best / better performance. From what I have read, is that it takes a lot of bandwith from the PCIE Lanes and this bandwith is limited. Am I right? If not, please enlighten me.
Also, is it allowed to use 4090 (consumer grade) hardware in business as hosting? Nvidia has special cards for those, with way waybhigher pricing.
Let them try to enforce legally where you choose to use physical goods you own. Might be a policy but ain't enforceable. Only exists for big players ie aws /Azure etc to force them to buy or they just withhold new allocation.
Nvidia stated in 2018 that consumer cards could not be used in datacenters but I don't know how well this is enforced in practice.
Does a VM have a dedicated GPU or is it shared? What sort of virtualization is provided? KVM? Panel included?
Yes, GPUs are 100% dedicated to a single user / VM. KVM virtualization with PCI passthrough. We have a panel with all of the features you'd expect.
Hourly billing is also available if you want to try it out: https://puregpu.com/
@PureVoltage ?
Hello @AAL,
At AccuWeb Hosting, we can provide a bare metal server with CPU, RAM, SSD, and much more. For more info, check out our dedicated hosting options.
This is something we can do but to be honest we're more so looking to not buy anymore 4090's and wait until the 5090's come out early next year as we just know half of the people wanting GPU's jump ship once something newer and far better comes out.
We build out a lot of custom servers for our customers in New York and other locations so feel free to hit us up for quotes.
https://purevoltage.com
That's the case for gaming, SLI and crossfire are not getting resources and they've been focusing on just making hardcore powerful GPUs. For crypto and AI workloads, the work is done ON the GPU and not over PCIe path as a bottleneck. So a lot of systems use PCIe x1 breakout cables.
Ah, right, @TimboJones. I was more thinking in a video encoding way. Not AI or crypto.
There are some ML / AI workloads that depend on having ample PCIe bandwidth. Typically only mining rigs are built on x1. Most cloud GPUs you'll find will run on PCI 3.0 x16.
What motherboard has 10 slots of x16? I haven't looked, but I thought 8 with a combo of x8/x16 slots was about the max without going hardcore expensive.
@TimboJones yeah usually 2 and max 8
I don't see why not, TensorDock's business model is centered around using consumer GPUs as servers.
Most AI workloads need bandwidth between cards. This is why NVLink exists only on datacenter cards, not on 4090.
For example in LLM inference for every token generated you need to transport layers through cards. Old ass Titan X uses 5GB/s of bandwidth in Tensor Parallel, so you basically need 3.0 x8/4.0 "4 for these cards.
https://www.reddit.com/r/LocalLLaMA/comments/1d8kcc6/psa_multi_gpu_tensor_parallel_require_at_least/