New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
The most cost-effective way to store daily increasing images?
Hi, recently I got a GPU server to generate images continuously. It will produce about more than 1000 images which is about 2GB+ every day. I am wondering what's the most cost-effective way to store these images. Some S3 compatible service is pretty cheap, but I am afraid it's hard to preview images. So if the best method is buy a storage VPS and run Cloudreve or other cloud storage service on it?
Comments
Store the parameters used for image generation, which is much smaller than the image itself.
Regenerate images on demand from the parameters.
@servarica_hani Have those "Unlimited expanding storage plan" with the downside that you need to reboot every now and then to get the 3GB/day addition.
Get hold of a Pixel 3 phone - unlimited storage in Google Photos at 16Mp quality forever for images uploaded from the phone.
/dev/null
Or just any phone with a magisk spoofer module
This will only work if we get Pixel 3 and upload using that device or it unlocks this feature for the account logged in the phone? How does this work? Sounds like good long term investment.
Also OP if these images are not very useful you can take risk (Server maybe deleted by Discord in future) and post in Discord channel as it has unlimited storage, my friend has uploaded TBs of data in Discord channel.
Link to his project: https://github.com/forscht/ddrv
I hope I am allowed to share the GitHub link.
Hetzner's managed Nextcloud (Storage Share). It's cheap, and you can preview the images in the browser using the web app.
You can do the same with telegram as well
you could use a compressed file format like .webp or .jpg if its not already in a compressed format.
Or you can buy unlimited storage from @servarica_hani (https://servarica.com/unlimited-storage/) and on cheapest package they are adding 3Gb of storage daily
Maybe you can store a low resolution image (64x64 px) and regenerate the full size on demand, if your pipeline permits, as a compromise between speed and storage. Otherwise, what I'd do:
rsynclocally from the gpu server on cronjob; hard drives are $20 per TBTry Cloudflare R2 s3 storage on a local JuiceFS mount. My write up and benchmarks https://github.com/centminmod/centminmod-juicefs 🤓