New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Tell Us About Your Self-Hosted (Or Rented) AI Rigs
raindog308
Administrator, Veteran
in General
I've been really pleasantly surprised at how well Apple Silicon runs AI models. My 64GB Macbook Pro runs 40GB LLMs models very nicely here on my lap. I phrased that sentence that way because at my age and financial station, that's the only way I'm going to get models on my lap.
I'm looking into getting access to an M2 Ultra Studio so I can run goliath-120B and maybe Meta's 400B if it comes out.
My gaming PC has a 4080 so that runs smaller models quite nicely.
I'm not doing any training of my own.
What are you doing with self-hosted AI?
Thanked by 1xxsl
Comments
I used Pinokio, this is an application that contains a lot of AI at once. But to be honest, I didn't really like it.
It's kind of shocking how good value the MBPs are now for at least casual use of LLM models, at least considering how low power they are in comparison to a dedicated GPU.
I've actually been using Colab/Kaggle but having access to beefy GPUs at work is making me want something more for personal projects, even if it's one of the services to rent a GPU for a while for training.
Even a 2020 M1 MacBook Air runs 8B models just fine, thanks to Apple's unified memory.
Do not praise apple dor performance! They will use that as leverage to increase price even more.
Good point. Though Apple being Apple they wouldn't wait to have any leverage and just do it anyway.