New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Looking for best and budgetable server for running Llama2-70b
hostcurator
Member
in Providers
We are planning to do some development works on Llama2-70b which needs GPU. For 70B models, we advise you to select "GPU [2xlarge] - 2x Nvidia A100" with bitsandbytes quantization enabled or "GPU [4xlarge] - 4x Nvidia A100".
Can some one here suggest me some providers and best plan with them for running this?
Currently we are on development stage. Once we are able to achieve our goal. We will be looking for production instance as well.
Comments
Hello @hostcurator you like RTX 2080 TI 11GB or similar GB of VRAM?It's possible rent you for very cheap price
Regards
You could take a look at TensorDock
Check out Crunchbits
https://crunchbits.com/gpu#Plans
Vultr has great GPU selection.
I've used their Nvidia A16 instance and it was the best offer online for my usecase.
Thank you very much for all your suggestions. Has some one tried Llama2-70b ?
Yes ... It's really subpar compared to gpt4 api
Hello, @hostcurator
You can check out our dedicated GPU servers
Hi,
Thank you very much for your responses. We were able to run and test the application on runpod.io. We were able to find our right resource now. Below configuration is needed for running our application.
AMD EPYC 7532 32-Core Processor
25GB RAM
1xRTX 3090 15GB VRam
500 SSD
Can some one will be able to provide quote for the same?