Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Storage solution question
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Storage solution question

pavspavs Member

We are a small bootstrapped ISP in a third world country where bandwidths are usually expensive and slow. We recently got a customer who need storage solution, of 10s of TB of mostly video files (its a tv station). The thing is I know my way around linux but I have never done anything like this before. We have a backblaze 3 storage pod casing which we are thinking of using as a storage server. The Server will be connected to customer directly so its not gonna go through the internet, because 100+mbps speed is unheard off in this part of the world.

I was thinking of using 4TB HDD all formatted with ext4 and using LVM to make them one large volume (50-70tb at least). So customer logs in to an FTP like client and dumps whatever files he/she wants. But the customer only sees a single volume, and we can add space as his requirements increases. Of course this is just on papers from preliminary research as i don't have prior experience with this kind of system. Also I have to take cost in to consideration so can't go for any proprietary solution.

My questions are:

  • Is this the best way to handle this probably, are there equally good or better solutions out there?
  • For large storage solutions (at least large for me) what are my cost effective options when it comes to dealing with data corruption and HD failure.

Would love to hear any other solutions and tips you guys might have. thanks!

Comments

  • I personally would not like to manage a single system of 70TB of data. Could you use something like GlusterFS (GPL) to create a file system distributed across many systems?

    However, my thoughts are:

    Weight the pro and cons. Does the company have backups, etc.?

  • @Silvenga said:
    I personally would not like to manage a single system of 70TB of data. Could you use something like GlusterFS (GPL) to create a file system distributed across many systems?

    However, my thoughts are:

    Weight the pro and cons. Does the company have backups, etc.?

    RAID might help here?

  • Void_Whisperer said: RAID might help here?

    Of course, I was proposing an alternate solution. I would personally never store all my data in one location, on one device. I was think more along the lines of the OS messing something up, and not entirely hardware issues.

    I would bet Amazon S3 uses a mere RAID0 for data storage. They are able to achieve 99.9999999999% data retention very cheaply by storing the petabytes of data across cheap and easy to replace hardware.

  • @Silvenga said:
    I would bet Amazon S3 uses a mere RAID0 for data storage. They are able to achieve 99.9999999999% data retention very cheaply by storing the petabytes of data across cheap and easy to replace hardware.

    Probably.

  • Go Ceph or Gluster or don't bother trying at all

Sign In or Register to comment.