AI server GPU recommendations

Nacorid

@pompompurin said:

@professorparabellum said:
i want to build a machine to run a local LLM. i run a minecraft server that sees around 1200 players peak and we get somewhere in the ballpark of 1 million messages daily which is impossible to moderate. We have an AI chat filter but it runs on gemini, and even though they have the cheapest tokens we would still eat through an astronomical amount on a daily basis.
What GPU should i go with for this? I was looking at the Nvidia T4 because it has very low TDP for how powerful it is and its around $1k. They also have the L4 which is twice as fast and is rated for the same TDP as the T4 but its alot more expensive.

i'd say switch to deepseek

No.

@SwimmingFrog said:
At stated above, you don't need a LLM for this. Yours is a binary classification task (predict toxic or not toxic). You don't need to generate any text, which is the expensive part of such models.

Check out for toxic classification models at HuggingFace https://huggingface.co/models?sort=downloads&search=toxic

With models based on BERT and optimized, you can classify a chat message in ≈1ms with a T4 or 10ms in a CPU.

This is the only right answer.