Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Storage endpoints - what do you expect and for how much?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Storage endpoints - what do you expect and for how much?

trewqtrewq Administrator, Patron Provider

Hey everyone,

Something that I've noticed had been a big thing lately is storage. People want it and they want it cheap. With that in mind the cheapest way to provide storage is not via a direct disk mount on a VPS but over the network.

What endpoints technologies would you like to see? How much space would you require and what's your ideal, (realistic) price point?

Thanked by 1pbgben

Comments

  • caracalcaracal Member
    edited June 2017

    I'd like to see something 3-4TB redundant, perhaps with throttled IO, 250mbit port. This could even work with just SFTP ~$10-15/month paid yearly?

    Would replace Gsuite for me.

  • $2 a TB with a SFTP connection

  • Europe kvm vps 200gb hdd raid(10or6) ~200mbps max 3.50€

  • 1 TB, sftp only. 0.001$ per GB ( I don't care if is raid 0 )

    Thanked by 1dedicados
  • hzrhzr Member
    edited June 2017

    a bit off topic but: I'm storing ~5TB entirely full of tiny files (majority under 150KB ea) that need to be read constantly. this seems to be terrible bc of metadata/overhead/something.

    what kind of endpoint would make this least terrible?

    i've tried remote mounts of nfs, moosefs, xtreemfs, tahoe-lafs, some fuse scp/sftp thing.

    also ~$1/TB

  • hzr said: a bit off topic but: I'm storing ~5TB entirely full of tiny files (majority under 150KB ea) that need to be read constantly. this seems to be terrible bc of metadata/overhead/something.

    Use compressed filesystems or databases. LZ4 compression for speed, or LZO for higher compression rate. Network/disk latencies are much higher than RAM to CPU latency. Eg: - http://www.excamera.com/sphinx/article-btrfs.html

    I had a scraper that stored webpages as compressed BLOBS into a database with URL as DB key. In your case, use the filename/path as a DB key. Or enable transparent compression in the DB.

    For a slightly more involved solution, use a hashing function on the filename to lookup a compressed bucket file. Read the compressed bucket and extract your file. Group together files by typical access patterns to minimize the number of buckets you need to touch for a given task. Compressing at the bucket level will give the advantage of block compression.

  • williewillie Member

    hzr said: ~5TB entirely full of tiny files (majority under 150KB ea) that need to be read constantly. this seems to be terrible bc of metadata/overhead/something.

    I thought S3-style object store was supposed to handle this well.

  • pbgbenpbgben Member, Host Rep

    So, to put this into perspective, its around 30-40USD per TB to buy storage. If the price is not reflecting that price then you will need to ask yourself how is it cheap? Limits on IOPS by using high capacity drives and filling it with users (10TB HDD with 10 1TB users on it will perform like a floppy disk)

  • williewillie Member

    pbgben said: (10TB HDD with 10 1TB users on it will perform like a floppy disk)

    Right, the idea is to only offer glacier-like interfaces instead of vps. That means backup users don't have to compete for iops with seeders on the same node.

    I don't think $0.001/GB ($1/TB) is realistic but $2/TB plus Glacier slowness and some steep transfer fees is available from both OVH and Online. Getting that with lower transfer costs would be great.

    At around $5/TB (old drives, no raid) you can get a Hetzner auction server (2x 3TB drives) so that's possibly a constraint in the other direction.

    I don't know what kind of hardware @AshleyUK is using but a Ceph cluster with commodity hardware seems like about the lowest cost per TB approach. I think he is doing something like that.

  • Personally anything with decent connectivity to Europe and NA. Asia servers are much more expensive and often leads to less storage so Asian speeds as long as 50Mbit+ I'd be happy.

    Location isn't an issue as long as what I've said above is met.

    As for endpoint and accessibility I'd like either some form of FUSE compatible method so I'd be happy with SFTP. I would like an S3 compatible storage as well for solutions that natively support is such as mailinbox ideally I'd like S3 to be billed in chunks of 10-50GB as I won't ever use more than like 50GB for S3

  • @willie said:

    pbgben said: (10TB HDD with 10 1TB users on it will perform like a floppy disk)

    Right, the idea is to only offer glacier-like interfaces instead of vps. That means backup users don't have to compete for iops with seeders on the same node.

    I don't think $0.001/GB ($1/TB) is realistic but $2/TB plus Glacier slowness and some steep transfer fees is available from both OVH and Online. Getting that with lower transfer costs would be great.

    At around $5/TB (old drives, no raid) you can get a Hetzner auction server (2x 3TB drives) so that's possibly a constraint in the other direction.

    I don't know what kind of hardware @AshleyUK is using but a Ceph cluster with commodity hardware seems like about the lowest cost per TB approach. I think he is doing something like that.

    HP DL servers, Juniper Switches, HGST Drives

  • OliverOliver Member, Host Rep

    Out of interest for those with some storage service already, what kind of data transfers are you typically doing? For example if you have 1TB storage VPS how much are you actually moving in and out each month?

  • @Oliver said:
    Out of interest for those with some storage service already, what kind of data transfers are you typically doing? For example if you have 1TB storage VPS how much are you actually moving in and out each month?

    A whopping 17GB this month on my 1TB VPS.

    Thanked by 1Oliver
  • NekkiNekki Veteran
    edited June 2017

    @Oliver said:
    Out of interest for those with some storage service already, what kind of data transfers are you typically doing? For example if you have 1TB storage VPS how much are you actually moving in and out each month?

    Typically, after the initial load up (between 2-4TB depending on what's being backed up), I'm doing a further 5-10GB a week in and about 1-2GB back out.

    Of course, at some point, I'll possibly want the 2-4TB back out (probably at a decent clip too)t, so the pricing for that level of retrevial is important.

  • @Oliver said:
    Out of interest for those with some storage service already, what kind of data transfers are you typically doing? For example if you have 1TB storage VPS how much are you actually moving in and out each month?

    Per GB I use around 1-2GB of bandwidth. If I need to restore it could be up to 3GB however id prefer to purchase additional than have 3GB per GB

  • OliverOliver Member, Host Rep

    Hmm I probably should have asked as a ratio. So in the case of @Dextronox ratio is 2% and even lower for @Nekki

    @GenjiSwitchPls do you mean Per TB you use 1-2GB of bandwidth?

  • @Oliver said:
    Out of interest for those with some storage service already, what kind of data transfers are you typically doing? For example if you have 1TB storage VPS how much are you actually moving in and out each month?

    for the last 2 year, 100 GB top.

  • Weren't we told that storage is nowadays dirt-cheap?
    I don't know why people still charge $5 per month for 20GB of online-storage.
    As for me, after I've seen that you can get 11TB of storage for 60€/month on a 1gbps server with 25TB of traffic (see seedhost.eu's storage boxes, ~184GB per euro), I don't see a reason why I should pay more than 3€ per 100GB, only talking about storage though.
    Traffic wise should be able to at least upload the storage once completely, better are two times, per month.

    I've moved on my 400GB VPS about ~ 2.6TB last month, with inbound traffic around same.

  • @Oliver said:
    Hmm I probably should have asked as a ratio. So in the case of @Dextronox ratio is 2% and even lower for @Nekki

    @GenjiSwitchPls do you mean Per TB you use 1-2GB of bandwidth?

    Nope per GB. I don't have a need for a 1TB storage VPS. I use 100-500GB for backups and they are wrote to daily or hourly depending on the data and data size

  • OliverOliver Member, Host Rep

    @GenjiSwitchPls oh OK, so you have a storage VPS and write 100-500GB per day to it?

    That sounds quite exceptional compared to the other replies.

  • @Oliver said:
    @GenjiSwitchPls oh OK, so you have a storage VPS and write 100-500GB per day to it?

    That sounds quite exceptional compared to the other replies.

    Wait. Am I tired what.i have a 100GB storage VPS and write around 100-200GB a month

  • OliverOliver Member, Host Rep

    ah OK, well still that ratio is quite different to the other replies.

    Anyway, interesting to read how people use these from a provider perspective. :-)

Sign In or Register to comment.