Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


RTX 4090 8x Bare Metal
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

RTX 4090 8x Bare Metal

AALAAL Member

Hi,

I'm renting a number of 4090 GPU BMs at the moment for a good monthly price but seeing if anyone else can offer similar pricing on a smaller deployment timescale.

The requirements are (if providing, e.g. 8x 4090s in a single bare metal server)

  • At least a 64 core CPU
  • At least 192 GB ram
  • At least 1TB SSD (raid not needed)
  • 8x 4090 GPUs
  • Out of band access
  • 9 Dedicated Public IPs
  • Location: preferably east coast US but US in general either

I would install Proxmox and create VMs giving flexibility.
It would be rented via a business so would need invoicing etc.

Thanks

Comments

  • LeviLevi Member

    Afaik, it is not possible to mount 8x 4090 on stock mobo. I asume you will pump those poor cards to 100% so power consumption alone would be humongous.

  • PureGPUPureGPU Member, Patron Provider
    edited October 15

    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

  • AALAAL Member

    It's mainly so I can manage the VM templates that I have everything installed on and able to get into the VMs via console. Do need Ubuntu 22.04 for example but I think you install Ubuntu 24.04 at least from the dropdown options.

    Right now I'm running all VMs with 1GPU so wouldn't be the too bad if you had Ubuntu 22.04 as an option.

  • PureGPUPureGPU Member, Patron Provider

    @AAL said:
    It's mainly so I can manage the VM templates that I have everything installed on and able to get into the VMs via console. Do need Ubuntu 22.04 for example but I think you install Ubuntu 24.04 at least from the dropdown options.

    Right now I'm running all VMs with 1GPU so wouldn't be the too bad if you had Ubuntu 22.04 as an option.

    We can definitely provide Ubuntu 22 and do custom templates, I'll shoot you a PM! We also allow users to mount custom ISOs.

  • AALAAL Member

    Great, thanks.

  • @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    I’m a bit of a newbie when it comes to GPU hosting, but I have always read that multiple GPU’d dont mean best / better performance. From what I have read, is that it takes a lot of bandwith from the PCIE Lanes and this bandwith is limited. Am I right? If not, please enlighten me. :)

  • LeviLevi Member

    Also, is it allowed to use 4090 (consumer grade) hardware in business as hosting? Nvidia has special cards for those, with way waybhigher pricing.

  • james50ajames50a Member
    edited October 15

    @Levi said:
    Also, is it allowed to use 4090 (consumer grade) hardware in business as hosting? Nvidia has special cards for those, with way waybhigher pricing.

    Let them try to enforce legally where you choose to use physical goods you own. Might be a policy but ain't enforceable. Only exists for big players ie aws /Azure etc to force them to buy or they just withhold new allocation.

  • @Levi said:
    Also, is it allowed to use 4090 (consumer grade) hardware in business as hosting? Nvidia has special cards for those, with way waybhigher pricing.

    Nvidia stated in 2018 that consumer cards could not be used in datacenters but I don't know how well this is enforced in practice.

  • @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    Does a VM have a dedicated GPU or is it shared? What sort of virtualization is provided? KVM? Panel included?

  • PureGPUPureGPU Member, Patron Provider

    @arvinzo said:

    @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    Does a VM have a dedicated GPU or is it shared? What sort of virtualization is provided? KVM? Panel included?

    Yes, GPUs are 100% dedicated to a single user / VM. KVM virtualization with PCI passthrough. We have a panel with all of the features you'd expect.

    Hourly billing is also available if you want to try it out: https://puregpu.com/

  • clay_pclay_p Member, Patron Provider

    @AAL said:
    Hi,

    I'm renting a number of 4090 GPU BMs at the moment for a good monthly price but seeing if anyone else can offer similar pricing on a smaller deployment timescale.

    The requirements are (if providing, e.g. 8x 4090s in a single bare metal server)

    • At least a 64 core CPU
    • At least 192 GB ram
    • At least 1TB SSD (raid not needed)
    • 8x 4090 GPUs
    • Out of band access
    • 9 Dedicated Public IPs
    • Location: preferably east coast US but US in general either

    I would install Proxmox and create VMs giving flexibility.
    It would be rented via a business so would need invoicing etc.

    Thanks

    Hello @AAL,
    At AccuWeb Hosting, we can provide a bare metal server with CPU, RAM, SSD, and much more. For more info, check out our dedicated hosting options.

  • PureVoltagePureVoltage Member, Patron Provider

    This is something we can do but to be honest we're more so looking to not buy anymore 4090's and wait until the 5090's come out early next year as we just know half of the people wanting GPU's jump ship once something newer and far better comes out.

    We build out a lot of custom servers for our customers in New York and other locations so feel free to hit us up for quotes.
    https://purevoltage.com

  • @DennisdeWit said:

    @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    I’m a bit of a newbie when it comes to GPU hosting, but I have always read that multiple GPU’d dont mean best / better performance. From what I have read, is that it takes a lot of bandwith from the PCIE Lanes and this bandwith is limited. Am I right? If not, please enlighten me. :)

    That's the case for gaming, SLI and crossfire are not getting resources and they've been focusing on just making hardcore powerful GPUs. For crypto and AI workloads, the work is done ON the GPU and not over PCIe path as a bottleneck. So a lot of systems use PCIe x1 breakout cables.

    Thanked by 1DennisdeWit
  • Ah, right, @TimboJones. I was more thinking in a video encoding way. Not AI or crypto. :)

  • PureGPUPureGPU Member, Patron Provider

    @TimboJones said:

    @DennisdeWit said:

    @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    I’m a bit of a newbie when it comes to GPU hosting, but I have always read that multiple GPU’d dont mean best / better performance. From what I have read, is that it takes a lot of bandwith from the PCIE Lanes and this bandwith is limited. Am I right? If not, please enlighten me. :)

    That's the case for gaming, SLI and crossfire are not getting resources and they've been focusing on just making hardcore powerful GPUs. For crypto and AI workloads, the work is done ON the GPU and not over PCIe path as a bottleneck. So a lot of systems use PCIe x1 breakout cables.

    There are some ML / AI workloads that depend on having ample PCIe bandwidth. Typically only mining rigs are built on x1. Most cloud GPUs you'll find will run on PCI 3.0 x16.

    Thanked by 1AXYZE
  • @PureGPU said:

    @TimboJones said:

    @DennisdeWit said:

    @PureGPU said:
    We have a 10x 4090 node coming online within the next week (hopefully). EPYC 7713, 512GB RAM, 4x 4TB NVMe, 10Gbps.

    However, we normally rent them as VMs, just for the convenience factor (easy reinstalls, backups, monitoring, RAID10 monitored and managed by us, etc). Is there a specific reason you need bare metal? We can rent either 1x VM with all of the resources and 10x GPUs, 10x VMs with 1 GPU each, or anywhere in-between. Nested virtualization is also possible.

    I’m a bit of a newbie when it comes to GPU hosting, but I have always read that multiple GPU’d dont mean best / better performance. From what I have read, is that it takes a lot of bandwith from the PCIE Lanes and this bandwith is limited. Am I right? If not, please enlighten me. :)

    That's the case for gaming, SLI and crossfire are not getting resources and they've been focusing on just making hardcore powerful GPUs. For crypto and AI workloads, the work is done ON the GPU and not over PCIe path as a bottleneck. So a lot of systems use PCIe x1 breakout cables.

    There are some ML / AI workloads that depend on having ample PCIe bandwidth. Typically only mining rigs are built on x1. Most cloud GPUs you'll find will run on PCI 3.0 x16.

    What motherboard has 10 slots of x16? I haven't looked, but I thought 8 with a combo of x8/x16 slots was about the max without going hardcore expensive.

    Thanked by 1Hajt
  • HajtHajt Member

    @TimboJones yeah usually 2 and max 8

  • gremeyergremeyer Member
    edited October 18

    @Levi said:
    Also, is it allowed to use 4090 (consumer grade) hardware in business as hosting? Nvidia has special cards for those, with way waybhigher pricing.

    I don't see why not, TensorDock's business model is centered around using consumer GPUs as servers.

  • AXYZEAXYZE Member

    @TimboJones said:

    For crypto and AI workloads, the work is done ON the GPU and not over PCIe path as a bottleneck. So a lot of systems use PCIe x1 breakout cables.

    Most AI workloads need bandwidth between cards. This is why NVLink exists only on datacenter cards, not on 4090.

    For example in LLM inference for every token generated you need to transport layers through cards. Old ass Titan X uses 5GB/s of bandwidth in Tensor Parallel, so you basically need 3.0 x8/4.0 "4 for these cards.
    https://www.reddit.com/r/LocalLLaMA/comments/1d8kcc6/psa_multi_gpu_tensor_parallel_require_at_least/

    Thanked by 1TimboJones
Sign In or Register to comment.