Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


File storage setup
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

File storage setup

So my file storage site project is coming along well but need some help with the storage side. My ideal setup would be a VPS that I can add block storage onto as and grow when needed (id planned a "slab" for this but not available in lux yet). Dedicated server is no good as can't grow storage easily. What would people recommend? Multiple kimsufi type servers clustered together? If so what filesystem? Lizards? Cluster etc? Needs to relatively easy to manage and add more nodes. Or some other way.

Comments

  • Are you not able to connect multiple storage servers to your frontend and then associate a node/filename with a user? (I remembered that YetiShare does this). Otherwise your storage costs are going to be rather large....

    There are a few companies on here offering block storage with vpses but they have limits.

  • I'm adding that functionality but servers will be in single location so wanted cluster.q

  • Does it have to be block storage because blob store sounds more useful for file storage, in which case you could pick any S3 compatible provider.

  • IkoulaIkoula Member, Host Rep

    If it is available where you're hosted i would ask for a NAS if i were you.

  • sibsoftsibsoft Member
    edited November 2018

    Sorry if thread is not actual anymore. But have few advices (we've started many file sharing sites):

    1. Never use VPS on a storage server. Disk I/O is very important. VPS will have these problems 100%.

    2. Never use software raid or LVM on storage server. The same reason - disk i/o.

    3. Several dedicated servers is the best option. With average download activity it's ok to have 8-16TB of storage space per 1GBps network port.

    4. Adding more servers should be done on business logic level, not on FS level. Keep it simple.

    Hope that helped.

  • Thanks for the info I'm already awaiting black Friday to give some dedicated servers/providers a try.

  • You can add drives to Hetzner dedis: https://www.hetzner.com/flexipack?country=us

  • I currently have cloud VPS with hetzner and would like one of their higher spec storage boxes if one comes up at a good price.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @lurch said:
    I currently have cloud VPS with hetzner and would like one of their higher spec storage boxes if one comes up at a good price.

    Are you dead set on it being in EU?

    Francisco

  • @Francisco said:

    @lurch said:
    I currently have cloud VPS with hetzner and would like one of their higher spec storage boxes if one comes up at a good price.

    Are you dead set on it being in EU?

    Francisco

    Yep if you had slabs at lux I'd be all over it.

  • lurch said: I currently have cloud VPS with hetzner and would like one of their higher spec storage boxes if one comes up at a good price.

    I see. If you're going to use it for a while, I think you have to just get one of the regular offers and pay the setup fee. The older ones do show up in the auction system but you end up paying more per month than buying the regular offer. Also now they are using 10TB drives in all the storage servers so they are a much better deal per TB than before.

  • mfsmfs Banned, Member
    edited November 2018
  • @willie said:
    Also now they are using 10TB drives in all the storage servers so they are a much better deal per TB than before.

    Have to warn that 10TB HDD is not good in terms of "hot" files storage. When filled 100% you may have poor performance because of a lot of seeks. However, that depends on type (size) of your files.

Sign In or Register to comment.