New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
To be honest, I would like to try a few models, I don't even know how much Memory the system has, but Qwen 3.8 27B Abiliterated sounds good.
You can manage then endpoint if you want, but you could also give a trusted user SSH, could compile llama.cpp from source and see what fits best.
It would run behind a proxy anyway, otherwise we couldn't give out keys.
Honestly, I don't think llama.cpp would be best here, llama.cpp is great for lowend GPUs and offloading but vLLM/SGLang would be fastest here. It has 64GB of VRAM, could run FP8 or NVFP4. Maybe even fit some quantized version of H3 too, could have on GPU for LLMs and one for videogen. But again, annoying to handle content on videogen.
I could consider giving you SSH but you'd have to promise not to do anything funny on it... have been burned too many times giving folks access to servers (both in cloud and "home"(colo)lab, lol)
And yeah obv. a lightweight proxy on top with limiting + keys. Wish LET has OAuth, smh
I wonder if I would need a patron provider tag if I gave away API keys to LET members for free for a hobby endpoint..
I already coded automatic verification for LET, people could request keys automatically with certain requirements. If you really plan to do this, this weekend, I could prepare stuff.
If its free, it should be fine, but asking is always good.
Well, I can try but I have been really busy lately, so I can't make any guarantees. I will follow up by DM when server is back up. If you already created a LET bot for verification that's great, when I bring server back up I can take care of setting up inference, etc
lol
I would have to look at the benchmarks, there are some custom build engines for the 3090 and 4090, no idea about the 5090 though.
RAM would be also interesting, turns out MoE learns to fly with fast VRAM and lot of RAM.
I wouldn't worry about that, I run a few projects, which are sponsored (in my signature)
True! But I think vLLM/SGLang is about as optimized as it gets
Both 😜
Ideally this project would help make models available which aren’t available from other providers, such as Obliterated and H3 etc…
Abliterated Qwen3 why not
Abliterated H3 you're giving me PTSD from when I used to run a public videogen endpoint fuck no 😭
Maybe H3 with safeguards...
Thank you @averagedatahoarder for helping to edge this dream towards reality
💀
I don’t dare ask…
Well, maybe LET community will be different 🙃
I run this:
https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED
Comfortable 30tok/s on M2, i can fit larger but the concurrent performance is not there.
Ah this is the one by the Pliny guy, he is a grifter. Would not trust his models 😂
He is good at hyping his stuff up though
You just described every ai influencer in existence
😂
I search models by "uncensored" and similar tags, i dont really look who provides them - for my use the more shady the better anyway
Personally i wait now for a Deepseek 4.1 flash model, i already run the official at work but i cant exactly plug my porn bots into that cluster (well, i can, but my boss will be not amused and bill me the tokens)
D> @William said:
No way are we running DSv4.1 flash, it's twice as large (total not active) as DSv4 flash and even DSv4 flash would be out of reach
yea only with SSD offloading on enough unified memory or with a 100GB card
Tldr went irl to day but server issues, brought server home and need to work on it a bit. Hopefully by EOW if i have time