Cheapest set up for a Qwen 27b model

Comments

• naas1201 Member

  November 8

  @sumo said:

  @naas1201 said:

  @sumo said:
  You could try a v100 or a mining gpu. Used mining GPUs are hit or miss though. Picked up a few mining GPUs to try out local LLMs, about 1/2 of the ones I got didn't work or stopped working after a few months.

  Yeah you are not the first to tell me that, I am going china chengdu next month and though of buying one (dirt cheap there) but i think i will not bet on it, too risky indeed.

  For the v100 and heatsink, I bought them used on ebay 1-2 years ago when they were very cheap for the 16gb non pci version. They're only used in servers and I don't believe are popular with miners, so they should be close to new if recent pulls. You can try buying from ebay sellers called "core" something and "rhino" something. They seem legit and I think one is on here. I picked up the pci adapter from aliexpress, used a dual case fan for cooling, thermal paste. Just make sure you have adequate power from your ps.

  You can also try the p40, but I don't believe power is ever idled on that.

  Nice ! I am in philippines right now so no Ebay and anyway every hardware is expensive in this hell of a country (all due respect to ph people that i respect a lot)
  For the P40 no bc too slow if i recall, that would make the token speed to max 15 tok a sec if i recall (i also checked it) but yes can run bigger models.
• BilohBucks Member

  November 8

  if you would have a little bit more RAM then you could use Strata https://github.com/Niko1221/Strata

  Thanked by 1naas1201
• naas1201 Member

  November 9

  @BilohBucks said:
  if you would have a little bit more RAM then you could use Strata https://github.com/Niko1221/Strata

  interesting, thanks for that. I am still comsidering upgrading ram and ssd inside. Ssd is easy, the ram is the hard part. In vietnam they were ok to upgrade my ssd but not my ram (they way i do it is to ask pro first then do it myself)
• averagedatahoarder Member

  November 9

  @BilohBucks said:
  if you would have a little bit more RAM then you could use Strata https://github.com/Niko1221/Strata

  Strata is a scam
• averagedatahoarder Member

  November 9 edited November 9

  @averagedatahoarder said:
  (Also if you are interested, am planning on starting a free LLM service for LET members w/ Neoon as soon as possible, but need a new server for the GPUs)

  Server acquired, already had GPUs, need RAM and then we are good to go

  Hopefully Qwen4 comes out in time. Would be fun.
• BilohBucks Member

  4:15AM

  @averagedatahoarder said:

  @BilohBucks said:
  if you would have a little bit more RAM then you could use Strata https://github.com/Niko1221/Strata

  Strata is a scam

  why you think like that?