New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Community GPU servers
Any kind provider have some GPU servers that aren't currently being utilised and wouldn't mind sharing them to run some AI models on for the benefit of the LET community? I know @Neoon wouldn't mind administering it and I'm sure we can put together a mechanism to limit to decent LET members and prevent abuse.
Comments
And who will cover the electricity bill for those providers, even if they have the hardware?
As I said, it would be a greatly appreciated contribution to the community.
Are you gonna contribute? Or do you just want to leech?
LOL
You put limits on each account, fair share.
Maybe if LET pool together the money for one.
GorillaServers
AMD Ryzen 7700 (8 Cores / 16 Threads)
64GB DDR5 RAM
1.92TB NVMe
RTX 3090 GPU
1Gbps Unmetered
/29 IPv4
Datacenter - Ogden, Utah
$199/m
GorillaServers
AMD EPYC 7443P (24 Cores / 48 Threads)
128GB RAM
1.92TB Gen4 NVMe
4 x RTX 3090 GPU
1Gbps Unmetered Port
/29 IPv4
AMD
EPYC
GPU Enabled
4 x RTX 3090 GPU
Datacenter - Ogden, Utah
$699/m
This is an interesting thought. How would that hypothetically be managed by the community? Seems like moderation and fair access would be a fairly big job as well. Just handing over GPU machines seems like it could go poorly without safeguards - one bad apple could spoil it for everyone else.
Let someone like Neoon, msatt or daffy manage it. They all seem to manage their free services just fine.
$199/m
Damn your power is cheap.
huh? you give out API keys, they go with a token cap.
Depending on demand we change these caps.
Cast a vote to choose which models we offer and we be set.
I have an unused M2 mac studio with 128gb RAM from work but no fiber or static IP here, a chatbot interface to like DeepSeek Flash or Qwen 3.8 uncensored/liberated would be interesting, really just add auth and user management to OpenUI
That's fine, VPN would do the rest, fiber isn't needed either, low bandwidth except when fetching the model.
I ditched OpenUI for llama.cpp so far zero regrets.
Do you think we could vibe code an UI for handling multiple llama.cpp instances? @Neoon
whole community depends on a 3090, so it would be 0.001 tok/s/user ?
Just to be clear, it is interesting but not saying we're going to be offering something up for free. Maybe a community discount on something, but someone would be responsible for billing / anything that occurs on the machine. Really depends on what kind of specs would be needed.
Plenty open source software is available, example:
OmniRoute can connect to local Llama models (via llama.cpp, Ollama, or LM Studio) by configuring it as a local provider and enabling private network access.
So be able to share API keys (possible to set the usage restrictions) and a few desktop applications available which will allow to do tasks.
I wouldn't really be interested in a community project, because I'm sure contention would mKe it very very limited. But if someone trusted, like Neoon, wanted to be responsible for being the main contact for a small group of paying users, I'd probably buy in. It would need to be a rig capable of running something powerful though.
wdym, there are already open source load balancers.
Yea, we gonna fit 50 people on a single GPU, maybe even one million, maybe even infinity.
Public endpoints are already so cheap, that it doesn't make sense, at least from my point, to make anything commercial.
But if you got spare hardware, older hardware, you are not using or not using anymore, pretty sure we can find a use for it.
My thoughts would be to do models that aren't readily available like Qwen 3.8 Obliterated and maybe a decent video gen model.
Was thinking more to manage users and provide endpoints
A let version of Zero GPU? That’s a great idea. Let me know if there are any further plans—I’d be happy to put my 5060 to use for it. After all, it’s just gathering dust while sitting idle.
I (well, Zcode) wrote a user interface for a chatbot and proxy it back to the mac studio with GLM 5.3, single stream like 60 tok/s but Qwen should be better
EDIT: amusingly i plugged it into Zcode and it wrote its own webinterface with no online access
Not a hosting provider but have a GPU node free in a colo rack with 2x5090. It's currently down (very jank setup 😂 did it on a budget so need to go IRL to fix it) but once I bring it back up I'd be happy to host a LLM endpoint on it.
Can probably run Qwen3.8-27B on it and sustain quite a bit of load. Obviously won't be GLM-5.3 or anything, but nothing like that seems to have been proposed. Running distributed llama.cpp would be even worse, though 😭
Qwen 3.8 27B would be lit.
Yeah> @Neoon said:
Yeah, I mean given that we now have Fable 5.1 and Astra it's obviously not SOTA but would be fun.
I've hosted this endpoint for friends ("friends") in the past and it ended up getting abused quite quickly (someone shared a key with someone else who decided to run their entire website's content moderation system on it, lol) so would need to sort that out without invading privacy. But could be a fun project!
I can't think of any better model to host atm, MiniCPM5-2B is incredibly good for its size but they haven't released a larger version yet /shrug
Expensive, will be slow as fuck unless it's just one user using it all 😂
Need to bring myself to drive an hour to the datacenter this weekend lol. If it had to be one, would folks rather have Qwen3.8-27B Abliterated or something like Minimax H3?
From past experience running videogen endpoints I feel like hosting H3 would get messy fast (gooner bait + potentially illegal content), so quite a headache to handle it all