All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Cheapest set up for a Qwen 27b model
I need an ablated Qwen for regular penetration testing. I believe an ablated DS is unnecessary and that Qwen 27b should suffice.
I am now being mature enough to care about wasting money and time,and wonder whether to choose cloud-based options or build my own hardware, with cost as the main factor.
My usage need is about 4 hours per day.
For cloud services, I am considering RunPod or MeshLLM? (RunPod would run the model, and MeshLLM would use an API, but I dislike MeshLLM’s KYC policy, nor do I trust its owner, to be clear.)
Regarding hardware, many poor YouTubers and AI sources recommend a 3090, but I think I can achieve the same result with older hardware and skill, such as a P100 16GB (yes, I know AI sources say no).
I am not wealthy, and I am truly concerned about the best choice here; I would appreciate help.
Thanks to everyone for your long support on this forum, and I will see you in the comments for the BF soon anyway!
Naas
Comments
Check out the level1techs forums. Those guys love discussing janky-ass local AI builds.
Mac? The 64gb ones with m1. Maybe higher entry point but you might have better resell value so overall could end up cheaper
Thanks, will check that
True for most apple product so i take your advise, but man i hate apple
If you have NVMe drives and low RAM on your local setup, check out Colibri as well, if you don't mind slower speeds
Hey happy to help, have a bit of experience with this. But first could you let us know what you want to run on it (e.g. coding, roleplay, etc.)? And by ablated DS do you mean abliterated DeepSeek....?
You do not have to use it, just put an endpoint and keep it hidden.
I don't recommend either of these, you will have to pay a lot and MeshLLM is a bit of a grift.
If you want to go the rental path better to go on Vast/Clore/Salad no KYC and it will be dirt cheap. (If you use Clore be prepared for a huge headache but it is cheap so what can I say 😂 and on Salad you won't get anything on low/lowest priority)
Not really the most cost effective way IMO
(Also if you are interested, am planning on starting a free LLM service for LET members w/ Neoon as soon as possible, but need a new server for the GPUs)
You could try a v100 or a mining gpu. Used mining GPUs are hit or miss though. Picked up a few mining GPUs to try out local LLMs, about 1/2 of the ones I got didn't work or stopped working after a few months.
V100 not really worth it tbh
What is '4 hours a day' because a shitty GPU might make so few tokens per second that it takes days to hit the same tokens generated as 4 hours on a good cloud service
at only 4h a day renting will probably be cheaper ngl
im thinking more renting a shitty gpu vs a cloud service that will let you subagent like crazy at hundred of tps per subagent makes 'hour' a really terrible measurement for this case
Because of speed or price or something else?
Thanks for your answer;
I mostly need it for malware analysis and hack the box tuto (advanced research on learning etc)
I do mean abliterated Deepseek 4.1 1017 which i find really cool, but i think its overkill for my need.
Generally speaking i also installed some 4b uncensored on some phone and i like uncensored version more, as i believe it 's generally speaking more capable. (except for photo generation, i learned the hard way to never generate "18 years old" looking people with self hosted image generation, if you know you know)
Try Bonsai2 27b its Qwen 3.8 27b but heavily quantized and impressively keeps %98 precent of its knowledge
Actually really interesting, i am looking at them now, how the hck did i not know them until now !! thnaks for that
I would love to use it, but i must restrain as my needs might be flagged and put you guys at risk. Much appreciated otherwise !
Yeah you are not the first to tell me that, I am going china chengdu next month and though of buying one (dirt cheap there) but i think i will not bet on it, too risky indeed.
O yeah you are right, my bad on that. I meant 4 hours a day of llm for 30 tokens sec min. My bad.
If you are actually only using 400k tokens total I question the point of renting an entire GPU tbqh
If you want to run locally, I've done the same. I use the models with same use case as you for pentesting (27B abliterated) on a 7840HS (780M iGPU) with 32GB of 4800mhz RAM. It's slow (~8.5t/s), so for penetration testing I usually leave it alone for a few hours.
CyberTiel models is faster (~20t/s) and you should try it. 3090 will give you more speed but they cost arm and leg now as well. IMO it's not cheapest, but it's yours.
If you want to run the cheapest, you can rent GPU hourly and use it there (fast), or a KS server with 32-64GB and memory would work, but it's really, really slow.
Unpopular opinion since you mention "cloud services",
I currently using Google Gemini Ultra 5x, which is 100$ a month, its come with 1000 colab credit which you can use to rent those GPU for llm,
I was using it for Qwen 27B NVFP4, its rock solid 120tps, usable...
The gpu cost 10credit for an hour, so you get 100 hours
On top of that you also get $40 Google Cloud Credit which you can use for API or for VPS,
the other crap are:
65$ / 5h limit for Antigravity
20TB Drive and Youtube Premium...
Not the cheapest, but it just work and give more benefit
If you got 8GB of VRAM and 64GB RAM, you might try Qwen 3.8 Flash Next instead.
No idea what existing hardware you got, but buying a 4090 or 5090 for 4 hours per day, costly.
For the v100 and heatsink, I bought them used on ebay 1-2 years ago when they were very cheap for the 16gb non pci version. They're only used in servers and I don't believe are popular with miners, so they should be close to new if recent pulls. You can try buying from ebay sellers called "core" something and "rhino" something. They seem legit and I think one is on here. I picked up the pci adapter from aliexpress, used a dual case fan for cooling, thermal paste. Just make sure you have adequate power from your ps.
You can also try the p40, but I don't believe power is ever idled on that.
Thanks for the feeback, make me feel good to know i am not alone on this path.
Damn i feel like slow is the only path here anyway, I think i should go with a 3090 maybe, i just don’t like to follow the hype, 🐑.
was brave to say that here.
I used a grey service (difmark) to have a 2Tb subscription for clean project and got small credit each month, if i recall i also indeed have a ridiculusly free vm with it. So it’s actually right what you say, the problem is that i don’t trust google at all, and i consider all network and anything touching them contaminated, from that i would feel dumb if data of my clients are leaked for a technical that i don’t master (i know i can create tunel and stuff, but damn really i dont trust google)
My main is a potato pc, last apple air 2015 4GB ram i5 that i converted to a linux distro. I mostly operate from cloud vps that could be considered potato vps too (🤣) I am really a poor pentester.
Note that this is bc i wanna stay leggit, i sometimes really wanna use my “findings”. i often have target that use literral beast to run stuff like age of empire 2…