Need 200 storage vps 2tb each.

cloromorpho

Lol

vimalware

For a large scale mirror operation, you're better off centralizing storage at some place like Hetzner Storage boxes : 3.99 Euro/TB-month for the 10TB plans.
(https://www.hetzner.de/hosting/produktmatrix/storagebox-produktmatrix)

Use multiple VPS with multiple threads to 'stream' the output to these central 'SAN's via a FUSE mount. Definitely have a RDBMS in there somewhere to keep records. (postgres!)

Split up crawl and storage to keep your project somewhat resilient.

It might take much longer than you think with big sites like Pornhub especially if they are employing intelligent traffic throttling for their viewer streams.
Porn industry is always at cutting edge of technology .

willie

vimalware said: large scale mirror operation

I thought of this as more of a datahoarder operation than a mirror. Thus the idea of owning all the hard drives and keeping them at home instead of online, once they are filled. Maybe you even want to think about Ultrium tape or the modern equivalent if there is one.

rm_

willie said: OVH Archive Cloud

Isn't that some sort of semi-offline storage with no instant access? As the OP said

my hobby with friends, and maybe a couple allowed to access the website.

So there will need to be a website, presumably with real-time playback access to files.

ngstargate

"Unlimited" Hetzner - https://wiki.hetzner.de/index.php/Traffic/en

If the traffic limit is exceeded then the bandwidth will be restricted to 10 Mbit/s.

willie

rm_ said:

So there will need to be a website, presumably with real-time playback access to files.

Depends on where the friends are etc. I mean there are always pornhub's own servers unless something happens. I always thought the concept of someone having 100s of TB of porn was a joke though. I knew a guy with 1TB and even that seemed like it would get hard to tell apart after a while.

telephone

@vpsman222 said:
look I don't care if this take 10 years to finish this my hobby with friends, and maybe a couple allowed to access the website.

Sounds exactly like PornHub. Teach them how to enter the website into a browser and you're set .

My suggestion: Download the videos you like and leave the rest on PornHub. They're not going anywhere. There's no need to waste hundreds of TB's on @Nekki's crazy hentai tentacle pony porn.

rm_

telephone said: They're not going anywhere.

Thaaaat's debatable, recently the largest anime/hentai tracker nyaa.se went down, luckily some old DB dumps have surfaced and people are already building replacements; another tracker, bakabt.me, went private-only. Also YouTube channels get closed down and deleted entirely for silly copyright shit all the time. Stuff does get taken down from the Internet every day, never count on anything always staying there.

pbgben

@vpsman22 use the SYS ARM nodes, the ones I tested had 500Mbits down (Capped to 10MB upload)

surfcu

@Rhys said:

telephone said: (when rclone was working).

bifrost :: ~ » rclone listremotes -l
acd:    amazon cloud drive
gdrive: drive
bifrost :: ~ » rclone ls acd:/ --max-depth=2
14229253355 EtqOfzIzs73X/VyWUOeOEHd3oY1N6VZLf
  1432589 EtqOfzIzs73X/cS6GIgcC9L-Owf3HXFsj4oHNTvdOLGyicx-H-5S3lMtpQKP6X-
44210657842 EtqOfzIzs73X/TGYC2jbVHnI29PlUbqA
17933608968 EtqOfzIzs73X/7c-,SB9W-rsZIf7DjIeGW0
 34064074 EtqOfzIzs73X/9fkAaz9yZrL7Kcudz62qNP5KrSA
  4544562 EtqOfzIzs73X/8QbMkoftyNKGIkh4Rke9s7LiWHXM1vLDXFnJPB646qRBBcwx01
bifrost :: ~ » rclone ls gdrive:/ --max-depth=2
 34064074 EtqOfzIzs73X/9fkAaz9yZrL7Kcudz62qNP5KrSA
17933608968 EtqOfzIzs73X/7c-,SB9W-rsZIf7DjIeGW0
  4544562 EtqOfzIzs73X/8QbMkoftyNKGIkh4Rke9s7LiWHXM1vLDXFnJPB646qRBBcwx01
  1432589 EtqOfzIzs73X/cS6GIgcC9L-Owf3HXFsj4oHNTvdOLGyicx-H-5S3lMtpQKP6X-
44210657842 EtqOfzIzs73X/TGYC2jbVHnI29PlUbqA
14229253355 EtqOfzIzs73X/VyWUOeOEHd3oY1N6VZLf

¯\_(ツ)_/¯

How did you manage to do it?