New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Local LLM Fine-Tuning on Raspberry Pi 5
Accidentally burned 3 SD cards trying to run Llama2-7B on Pi5 8GB variant. Finally got 4-bit quantized model working at 2.8 tokens/sec using llama.cpp with AVX512 optimizations. Pro tip: Attach copper heatsinks to RAM chips when doing 72hr+ training cycles. Anyone else experimenting with edge-device AI?
Comments
Why train on a Raspberry Pi? Google Colab seems way better, right?
Maybe that's the case, but I enjoy the fun of tossing more.
to fine tune do we need a file json formatted questions and answer? or can we feed docs and it can be done?