New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Anyone have suggestions for inexpensive large-scale object storage?
gamingarchive
Member
in General
Does anyone have suggestions for large-scale object storage? I'm looking to host some backups of a few hundred TBs of public-interest data, but my budget is pretty tight...
Currently I'm (ab)using the free tiers of Modelscope and Modal (both VC-backed AI companies) that offer unlimited storage, but I'm unsure how much longer they will offer unlimited storage so I'd like to find an inexpensive backup option I could pay for.
If anyone has suggestions, I'd really appreciate it 
Comments
Backblaze but a few hundred TBs of data won’t be cheap anywhere unless you’re abusing something somehow
Thanks! Yes I've been looking at Backblaze... R2 is way too expensive for my usecases, I was wondering if the Personal Storage system could be reverse engineered to use it as object storage rather than a mirror for an existing drive but this would probably go against ToS...
You'll be hard pressed to get object storage at lower prices than AWS Glacier Deep Archive. It's $1 per TB. Other providers struggle to provide even raw storage at those prices.
Anything cheaper than that and you are abusing the ToS.
For reference, you can't even do it yourself for under $1 per TB per month using hard drives.
HDDs are $20/TB which means safely storing 1 TB costs $1 per month just in disk space and even then you only break even after three years. In addition you have to pay for the NAS and power, which further extends the payback period.
Tape drives aren't much better, since the drives are so expensive. LTO tape storage only becomes competitive after 300 TB or so.
Fair enough. I guess I am just being delusional here then haha
@gamingarchive literally archive.org
Does modelscops login even work?
Yes, it worked for me. They have two sites, .ai and .cn, one is US and one is China I think. Both have terrible upload speeds from the US
I asked, they will not take my data. Too risky for them
Given their response, it's likely your data to be ROM or ISO?
I'd recommend building windows machine + backblaze + nearline/ultrastar HDD. Give or take, it would cost $2000 plus $3000 for every 100TB plus $99/year. So, 500TB costs around $17000 at the first year. The second choice would be to find data hoarders (other than AI data provider)
Backblaze will really store 100TB in their "unlimited" $99/year plan?
Yes, so long as you have all the data on a disk somewhere. I suspect most people use much less, so it's subsidized
Yes, data is in that direction haha
I've worked with datahoarders in the past but was hoping to be able to do something of my own without spending an insane amount of money. But thanks for the suggestion I will look into that
Read this yesterday: https://rareese.com/posts/backblaze/
(Their TLDR, not mine.)
Is it possible? Yes.
Can everyone do it? Kind of. Scaling beyond few TB require optimization and resources. There's all kinds of silly bug
https://old.reddit.com/r/DataHoarder/comments/1c89ijt/43tb_of_data_backed_up_to_backblaze_in_2_weeks/
I have about 25TB stored. On personal plan: No linux, no NAS, no servers, no vmdk/iso/qemu/qcow/etc, no windows server editions allowed.
This is at least a little bit reasonable given how badly file stubs and VFS interact
All this data must be attached to a single computer for this to work?
Yeah. I have 4x16TB SATA in my dev machine.
I know someone has reverse engineered Backblaze's API to be able to programatically download files. I wonder if it would be possible to "spoof" files on the drive to programatically upload files / use it as object storage. Though this would likely go against ToS/result in a ban...
There's other way to do it. But, it's uncertain if your files could survive for more than a year. Or, you may face problem with data retrieval.
Iykyk
That's the popular cloud backups. So, it's kind of grey area since it's technically the mounting point of network storage but it's still within the local storage. That's why I'm looking for cheap LTO machine.
Wait, Is this to copy the game site that is shutting down because of people trying to sell access to a free piracy resource?
Not quite, feel free to PM me and I can explain
If you'd be willing to share, I'd love to hear
Other backup solutions I found besdies Backblaze were Carbonite and CrashPlan but these seem pretty similar
1fichier is really nice. With premium gold subs, it's possible to treat it as delay memory, but it's not reccomended.
Let's say you found a file-sharing website similar to https://fumo.me. It's very unlikely it has capacity beyond a TB and the owner may delete it anytime since they found someone's junk occupying large amount of space
Yes, the free file hosts I have used before offer this sort of storage but I really want to avoid using them, the experience is pretty terrible...
This whole thread is basically why we can't have nice things.
Sorry
Perhaps a different but related question, and if I am posting in the wrong place / should open a new thread instead please do let me know:
Does anyone have recommendations for an inexpensive place to dump 10-20TB temporarily? Don't need root access but ideally some sort of shell, so not S3.
I have a seedbox through Seedhost for ~$15/mo for 7TB, are there any better deals than that?
@PulsedMedia has €9.99/1TB/y storagebox
There's 7 extra months if you play with its AI support Väinämöinen (e.g by creating tickets), if you bought the deals on the thread, posts your order id and write 2 paragraphs of your experience with it. But it also applies to monthly deals
However, if you really want to have the complete array, consider having title that says something like this
Looking for servers to host 500TB data. In the past, some hosts could offer $1/TB/m deals with >64GB of ram per machine.