New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
CPUs are not efficient for LLM, it can be done, but only with "small" models (4B, 8B) and rather slow. For good performance you would need a GPU with lots of RAM (great timing!) which is going to be expensive.
So i would suggest not hosting it yourself but using one of the many APIs like Openrouter for the LLM Part.
If you are just testing and playing around, Hugging Face would be a good place to start.
The Pro plan costs $9/month and will give you a daily quota of 25 minutes of half H200 GPU (time counted only processing the requests).
It is not much, but other than that the cheapest I could find when I looked into it was $0.50/hour, and that was for the instance being on, not actually the processing time.
KS-LE-B, dedi with 64gig, can be abused 24/7, nobody is gonna complain.
on the VPS the other hand, you might run into abuse problems.
Besides that, CPU only isn't going to be fast, prob cheaper just to a GPUs with 16GB+ VRAM.
If you only need it for a few hours at a time, I quite like vast.ai
If you are not yet a hardcore user, Hugging Face Pro subscription has 2million tokens monthly on some providers:
Disclaimer: I don't subscribe this service, ended up using my desktop GPU for my testing (RX 7800XT) - less processing power, smaller models, but it was already paid for.
in my own production, 4x3090 x 2. just some coder models along with it's embedded/reranker stack. being used for ~40 ish people.
if you're just starting / learning to deploy LLM, use free service that hands out credit to deploy gpu server like modal.com. if you're willing to pay take a look at runpod / salad.com
rent them gpu by hours. if you trust openrouter enough, go with them. https://openrouter.ai/docs/guides/features/zdr
don't ever get into BigGPU purchasing without proper planning (especially your in/out token spending). well, unless you can buy them at MSRP i guess. see https://www.latent.space/p/gpu-bubble
if your client really need confidentiality and can't use external resources. make them pay for their own gpu server. i manage few H100 cluster like this for banks / private corporation.
CPU inference really going to crawl once you have a lot of context (like beyond 100k).
imo using huggingface pro for inference doesn't seems to be efficient. 2mil token is nothing (ymmv)
I tried to run LLama on my local dedi on E5-2660v4 with 64GB RAM and it was very slow. I had to wait c.1 min for an answer to appear.
I've been running self-hosted Llama models on my own VPS for about a year now. Honestly the biggest factor is RAM – you need at least 16GB for anything useful, 32GB+ if you want decent speed.
I use Ollama – easiest setup I've found, works well on Ubuntu with Docker. For cheap VPS with enough RAM, look at Hetzner or OVH – better price/RAM ratio than most providers.
The catch nobody mentions: LLMs hammer your CPU between requests and memory leaks are real if you run them 24/7. Worth monitoring closely.
Bro, it is Friday, what do you expect, LLM is on vacation, you were lucky he replayed you in 1 MIN.
But yes, x86 is not the way to go.
@idroid007 - it will be much much cheaper to use a subscription based LLM at the moment, there are plenty you can choose from, let this be the problem of companies that have big pockets to perfect, don't waste your time and $$ for the moment on a technology that is basically in infant state and is as efficient as a 5.3L V8 Corvette from the 70's.
It was more like an experiment. Def not good to be run on a VPS.
I got the impression that Hugging Face pro would run on half nVidia H200.
What was the use case on that $6 tokes usage?
testing mcp servers that we create to make sure it really works with the recent / popular LLM for the tool calling process.