Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

300TB of online storage on a budget

124»

Comments

  • kevindskevinds Member, LIR
    edited February 2024

    @jon617 said:
    have you also considered the time it would take to upload that much data? Distance and quality of the backbone may be a factor, for example, if the provider has good peering and is a couple cities/states/provinces/countries away where you can get 100 Mbps uninterrupted, it would take 35 weeks to upload 200TB

    It is already 'in the cloud', one would just 'download' it from Dropbox at whatever speed the hosted server and Dropbox can transfer at. Can you download from Dropbox at 10 gbps?

  • jon617jon617 Veteran
    edited February 2024

    I think most (if not all) cloud file storage providers cap throughput per user, to save on bandwidth and disk usage, both are limited resources. Try it - see how fast it will be.

    I think any provider that has no overage fees and knows what they're doing, including storage, vps, and even datacenters, would always have some resource caps in place so one customer doesn't degrade the service for others. Just ask a datacenter or provider what they consider a 'DOS attack' to be, and why they don't like it.

    One personal experience was with Google Fiber's 1gbps upload & download to the home, while I truly had 1gbps to the net and the latency to Google was excellent (all google services were super quick, especially noticeable with ultra-quick seeking on youtube videos), there was definitely a speed cap when uploading and downloading to youtube, google drive, photos, etc. It varied by service, and I think it was never faster than 200mbps. That was years ago, so it's possible policies/resources may have changed since.

    I think larger providers may also have more technical concerns over just per-user policy limits, like if data is compressed/decompressed or encrypted/decrypted or moved around within their network before it can be accessed, there may be disk, CPU, and other factors. I would also imagine when dealing with many TBs of data that some requests could even fail, where a file isn't available because a resource is down, which could further slow download efforts.

    Curious what would happen if you buy all the hardware, and pay all the fees to prep the new online storage, only to realize dropbox has some ridiculous cap giving like 30-50 mbps? If that would happen, while 50mbps is sufficient to stream 4K HDR video, it would take a year to download 200TB of data.

    Thanked by 1rtsh
  • @jon617 said:
    Curious what would happen if you buy all the hardware, and pay all the fees to prep the new online storage, only to realize dropbox has some ridiculous cap giving like 30-50 mbps? If that would happen, while 50mbps is sufficient to stream 4K HDR video, it would take a year to download 200TB of data.

    Dropbox download is between 10-20MByte/s (80-160Mbps), even if you have 1G link.
    Would take 100 - 200 days to transfer all files.

  • kevindskevinds Member, LIR

    @lowendtalkxdax said: Dropbox download is between 10-20MByte/s (80-160Mbps), even if you have 1G link.

    Even on paid accounts?

  • PUSHR_VictorPUSHR_Victor Member, Host Rep

    I don't think OP will be able to move this data to anywhere. It will take forever regardless of uplinks. Encoding to HEVC and then moving will take 2x forever again, regardless of GPUs used if any are used at all. Still if budget is >$1000/m, I would be willing to help with S3-compatible object storage with nine nines durability. PM me.

  • edited February 2024

    @kevinds said:

    @lowendtalkxdax said: Dropbox download is between 10-20MByte/s (80-160Mbps), even if you have 1G link.

    Even on paid accounts?

    Dropbox Business, according to multiple sources.

    Here some test results from users:
    https://testmy.net/host-history/dropbox

  • vjunkievjunkie Member
    edited February 2024

    @lowendtalkxdax said:

    @jon617 said:

    Dropbox download is between 10-20MByte/s (80-160Mbps), even if you have 1G link.
    Would take 100 - 200 days to transfer all files.

    Not sure where you got this. I have 5gbs fiber and can download or upload at 3gbps speeds both directions. If I do a transfer from DB to another provider, it fluctuates. Db will give me 3gbps down but the other company might cap upload at 1gig.

    There were only a few storage servers that I could actually pull the full 5gbs download, Google Drive being one of them, but your limited to 10TB/day.

  • @jlet88 said:
    I would NEVER EVER trust 200-300TB of my most precious family memories in the cloud, anywhere, by anyone, unless I also had TWO robust local copies in my physical possession. The 3-2-1 backup strategy is the only way to go for something so important.

    I mean, seriously.... this is awesome that you have this special content, congrats to you for collecting it! This content is priceless to your family for generations. "Pictures of family albums that have been digitized, and lots and lots of family vhs recordings transferred to digital going back to my grandfather getting his first old school portable vhs recorder and filming his family going everywhere."

    Way too precious content to be screwing around with. I'd personally break them down into more manageable chunks (50-100TB per chunk), and set up a few UNRAID servers or Synology servers with proper redundancy for your local copies.

    This will cost you money, of course, so it's worth talking with your family to share costs. It is worth it!

    Additionally, I'd look into transcoding ALL the videos into a more compact codec, if possible, to save space!

    Once you have your own LOCAL, REDUNDANT (TWO COPIES) set of data IN YOUR PHYSICAL POSSESSION, then I would really figure out with your family (among other like-minded smart family members), so you can establish a long-term sane plan for the family to help cover costs and retain a minimum 3-2-1 standard of data protection (3 copies of data, 2 copies local, 1 copy off-site). The off-site copy can be a cloud service (expensive) OR it can be another bunch of hard drives (or UNRAID/Synology servers) over at a trusted family member's house.

    Good luck! You have a treasure! Take good care of it!

    as someone who has a small hom server and backups accross the world and as someone who lost picutres many many years ago, i can only agree

    Thanked by 1jlet88
  • SmigitSmigit Member
    edited February 2024

    @jon617 said:

    @jon617 said: This is my personal backup of over 20 years of "stuff".

    Also, I am assuming you're looking for plain storage, not backups. A 'backup' means you have data in one place, and want to make a copy elsewhere for online access and/or copy in case of data loss. It sounds to me you're looking to keep everything outside of the home, so if that were me and I needed everything hosted, I would buy or build 2 servers, put each in separate datacenters, and keep the two in sync. Run checksums and disk checks on a regular basis. A parity-based RAID, and keeping everything on one box, are both too risky with that much important data, in my opinion.

    This was my thought too. If the data is this important, I’d want 2x copies of at least the parts I don’t want to lose. The other copy could be local backups at your home, or could be a second co-located server. I would look at using different locations and different providers if co-locating in case for some unfortunate reason you lose the hardware and can’t recover the data. Maybe it’s some sort of catastrophic incident with the hardware beyond losing a few drives. Maybe the company goes bankrupt and doesn’t ship it back, or in shipment the hardware is lost or damaged. Dunno, but I’d want a second copy as there’s very few companies I’d afford the sort of trust that maybe Dropbox, Amazon etc would entail. While I didn’t follow it closely, the Dedipath closure seemed like a nightmare for those co-locating through them when they closed on short notice.

    But as others said, I’d be looking at trimming the data down too if possible. Not just to save costs. Sounds like at least some of this has some family importance, but if you become sick, hit financial difficulties or pass away I’m curious if a next of kin can cope with 300TB of digital legacy. I think most people would be completely overwhelmed by it. Would they continue to preserve that or cut ties with it all because it’s just too much to sort through?

  • You said somewhere 300-500€
    Imo you should well

    Re-encod3 most of your atuff and cleam it up
    Assuming now you have 100-200TB
    (Which is nuts still)
    You should.just use 2 homes with harddrives or tapes
    Datacanter is too expensive

    Im still unsure how you cam end up with 300 TB of VHS

  • @lowendtalkxdax said: Dropbox download is between 10-20MByte/s (80-160Mbps), even if you have 1G link.

    Would take 100 - 200 days to transfer all files.

    With rclone I was able to upload/download 50 concurrent files fully saturating my 10G link.

    Thanked by 1kevinds
  • AthorioAthorio Member
    edited February 2024
Sign In or Register to comment.