New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
30TB Storage
bitarchitect
Member
in Help
Hello LET!
I have about 24TB of videos (educational), sensitive data so security is very important.
Currently, we are developing an application which will use these videos later (paid service) . But for now we need to find a secure place for storing them until the app is out (we'll not remove them, we'll still use it as backup later).
What do you suggest for storage?
To buy a NAS and store them locally, or there is a secure cloud option to store?
Comments
yes
you can also let @emgh watch .. he needs more education
If it's sensitive, then you answered it yourself. But don't forget the RAID.
If you go that route, get something semi-pro, but please not the cheap ones.
RAID is not Backup, keep that in mind.
Try also Cloudflare Stream, it's more convenient and cheaper.
If the videos are critical to your business and you're a director (it sounds like you might be), one of the safest things you can do is get 2 (or more) NASes, have one copy at work and another redundant copy in your house. If you have other directors, let them have a copy too. These works especially well for "very cold storage" where you won't need to access the data, but if a catastrophic event happened to your office, you could still rebuild the company from the copy you have at your house.
By all means, encrypt the drives if it's sensitive, but make sure you have a copy of the key in multiple places too.
Personally, I'd also buy a large storage solution from e.g. Serverica, Hosthatch, etc, and borgbackup the data onto there. That way it'll be encrypted in storage, so safe from theft, but again make sure you have multiple copies of the key you use. You might not get 24TB in one storage device, but you can partition the data into appropriately sized chunks instead. Again, I'd have multiple of these, but it's your choice.
One thing I used to do with my personal data was the first one in reverse - I had 2 encrypted external drives, one always lived at work, one lived in my car. Every few days, I'd take the one out of the car into the house, do another incremental backup onto it, and put it back in the car. Next day, I'd swap it with the drive I kept at work. That way, I had 2 backups about every week, easy access to the older backup if I just did something stupid like deleting a file, and could get the most recent backup next day if required. Nowadays, I still have some backups on external drives, but everything is backed up daily to at least 3 different storage VPS running borgbackup.
Buy a couple (these days could be even a single) of large hard drives and keep a copy there.
If the data is important, consider having three copies, at least one of which is in a separate location.
3-2-1
three copies
two types of media
one off-site
If you don't want to retake those videos,
1. download them to a local drive
2. upload them to cloud storage
Hi,
you will always need a backup of your data. No matter if you have a copy online on some online storage ( cloud / what ever ).
So since you will anyway need to have a local backup, and since you are not yet production/live-ready -- buy some local NAS that you attach via USB ( 4x disk, maybe with build in raid ).
Like QNAP TR-004 or similar.
As soon as you are ready to spread your love to the world, you can upload it ( and still keep copies local of course ) -- dont ever dare to trust any provider, no matter the size. The data are your core business assets. Loosing them is like loosing most ( if not all ) of your work.
IF you need to upload the data, then make sure they are encrypted using a private key on your computer ( nothing that comes from the provider -- otherwise its simply not secure and private by definition ).
For really cheap it has to be: Tape, Tape, Tape...
LTO5 or better... Because of open file system used.
Used drives are cheap (new is not that bad if for a business) and you can get tapes for cheap.
24TB can easily fit on 17 LTO5 tapes (1.5TB uncompressed per tape).
So cheap... I would duplicate it 3 times and store the 3 sets in 3 locations.
You could also keep the 24TB on multiple hard drives, but a single new 24TB drive will cost more than a whole LTO5 Fibre Channel setup. Ask me how I know...
You don't really need a NAS as you don't plan to access the files on the network (yet). You don't need to store these online (and it would be risky).
Get enough xTB drives. Store several copies offline, in different locations. Problem solved.
If you need your backup strategy to survive a bombing of your country / region, send a copy abroad OR upload one online on the other side of the world (it's gonna be costly, though).
Personally, I wouldn't use tape as your main backup strategy. Maybe things are better now, but it used to be common for heads to drift slightly and you'd end up with tapes that'd only read on the drive they were written on. If the tape isn't used for a long time, it can stick and get damaged. Also, it's good for a daily backup where you want lots of backups, and your dataset for the day fits onto a tape (admittedly, you could do different data each day). But for a one off copy of a lot of data, not so much.
A few years ago, I had cause to read a whole load of QIC tapes from about 20 years before. For a few hundred tapes, we went through 3 drives, and had about 66% recovery rate. Also, reading from tape is sloooooow. I know these modern tape drives are probably somewhat faster and maybe more reliable, but unless you're regularly verifying these tapes you don't know for sure.
I'd say you want this kind of backup on regular SATA disks that you can plug into any old PC and know it'll just work.
Interesting, somebody gave you the setup for free? 12TB 7200rpm drive costs less than $100 new these days. LTO tape alone costs > $20 per cartridge, which is $340 total.
In fairness, that was probably true before all the chia mining nonsense started. Nowadays disks cost more than that again...
Just googled again, it seems the price went up while I was not looking
. But you can still get sub $100 renewed with 3 year warranty e.g., https://www.walmart.com/ip/MaxDigitalData-MD12000GSA25672-12TB-7200RPM-SATA-6Gb-s-256MB-Cache-3-5-Internal-Desktop-Hard-Drive-3-Years-Warranty/999436090?wmlspartner=wlpa&selectedSellerId=1386&gQT=1, which seems reasonable for archive if we're comparing to tapes, i.e. doesn't need to accessed constantly.
Internxt have a 10TB lifetime option... With the usual caveats, one of the backups could be buying three and uploading the videos there.
What you need to be doing is taking responsibility for the security yourself. That means encrypting them yourself before sending them to a backup site.
No... I bought the drives on ebay. 4x LTO5 FC drives, for just over the cost of the 12TB drive you mention.
There are companies that you can buy certified low usage tapes from... Cost me less than $3 per tape. The one I bought from do life time of tape warranty. They will just replace the tape.
My use case: just use multiple tapes, to have multiple copies on... Their cheap.
Data as so:
Live data: SSD/NVMe
Backups: HD - turned off when not in use.
Longterm backup/Archive: Tape. <-- what I'm building out now for myself (have used tape professionally)
Pixeldrain offers cheap storage
QIC is from the 1970s.
Things have changed a lot since then.
LTO was introduced in the 1990's. That's 1st Gen...
As I said, I wouldn't used anything before LTO5 - which came out at around 2010.
LTO tapes are rated for 30 years of storage, when stored in the correct environment.
Thanks a lot for your answers guys!
As I assumed, it's too risky to put it on cloud for now. I'll grab few new 12TB+ HDDs and store the videos on them (probably duplicate and put the disks in our two offices, different location).
I don't need access to them until our app is done. NAS is not needed for now, but if we'll need it, it is easy to get one.
Thanks again! We can lock this topic now.
Nothing is stopping you from encrypting them before uploading them to the cloud wherever.
The best solution seems to be self-hosting. Seagate has 24 TB HDD available. If you want reliability, you can store it in multiple places. For example, finding a storage server to store a copy of the data. Before uploading, you can encrypt the data using AES. Alternatively, you can use encrypted LVM on the storage server to encrypt the entire disk.
The problem with self hosting is people get sloppy and although they make 3 backup copies they store them in the same room which doesn't help when the room catches fire or someone breaks in and steals stuff.