All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Cheapest set up for a Qwen 27b model
I need an ablated Qwen for regular penetration testing. I believe an ablated DS is unnecessary and that Qwen 27b should suffice.
I am now being mature enough to care about wasting money and time,and wonder whether to choose cloud-based options or build my own hardware, with cost as the main factor.
My usage need is about 4 hours per day.
For cloud services, I am considering RunPod or MeshLLM? (RunPod would run the model, and MeshLLM would use an API, but I dislike MeshLLM’s KYC policy, nor do I trust its owner, to be clear.)
Regarding hardware, many poor YouTubers and AI sources recommend a 3090, but I think I can achieve the same result with older hardware and skill, such as a P100 16GB (yes, I know AI sources say no).
I am not wealthy, and I am truly concerned about the best choice here; I would appreciate help.
Thanks to everyone for your long support on this forum, and I will see you in the comments for the BF soon anyway!
Naas
Comments
Check out the level1techs forums. Those guys love discussing janky-ass local AI builds.
Mac? The 64gb ones with m1. Maybe higher entry point but you might have better resell value so overall could end up cheaper
Thanks, will check that
True for most apple product so i take your advise, but man i hate apple
If you have NVMe drives and low RAM on your local setup, check out Colibri as well, if you don't mind slower speeds
Hey happy to help, have a bit of experience with this. But first could you let us know what you want to run on it (e.g. coding, roleplay, etc.)? And by ablated DS do you mean abliterated DeepSeek....?
You do not have to use it, just put an endpoint and keep it hidden.
I don't recommend either of these, you will have to pay a lot and MeshLLM is a bit of a grift.
If you want to go the rental path better to go on Vast/Clore/Salad no KYC and it will be dirt cheap. (If you use Clore be prepared for a huge headache but it is cheap so what can I say 😂 and on Salad you won't get anything on low/lowest priority)
Not really the most cost effective way IMO
(Also if you are interested, am planning on starting a free LLM service for LET members w/ Neoon as soon as possible, but need a new server for the GPUs)
You could try a v100 or a mining gpu. Used mining GPUs are hit or miss though. Picked up a few mining GPUs to try out local LLMs, about 1/2 of the ones I got didn't work or stopped working after a few months.
V100 not really worth it tbh
What is '4 hours a day' because a shitty GPU might make so few tokens per second that it takes days to hit the same tokens generated as 4 hours on a good cloud service
at only 4h a day renting will probably be cheaper ngl
im thinking more renting a shitty gpu vs a cloud service that will let you subagent like crazy at hundred of tps per subagent makes 'hour' a really terrible measurement for this case
Because of speed or price or something else?