All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Dropbox alternative with robust sync capacity for a large number of files
Hi all,
I know this topic had been discussed several times on this forum, but they are too generic to address my problem, so I decided to start a new thread. As many others here, I want self-hosted solution to replace Dropbox, but strongly focused on sync capacity.
I want to sync about 400GB, mostly code, git, dataset, and photo - no video or large-sized files, which means huge number of files to index.
Now, I looked up for several threads in this forum and reviews online and found that non of the reviews are written based on my scale of sync need. Some reviews that claimed to test a lot of files were tested at around 20GB scale. Yet, I think I got a general sense of the performance.
Dropbox: expensive, nsa, but the best sync capacity.
Owncloud: may hiccup on big amount of data, slow sync client, great web client
Seafile: seems like it has good, fast sync capacity - can anybody share experience on syncing many files?
Syncthing: good sync capacity but no native support for Windows
pyd.io: not enough reviews
btsync: great sync capacity but closed-source and 10 folders limit
Considering my need, I'm leaning toward btsync + owncloud web interface + encfs. Sync capacity, reliability, and data integrity is more important than NSA to me.
Could somebody share experience with any of the open source solution with huge number of files to sync like what I need? Or any suggestions appreciated.
Comments
Regarding owncloud, I've got over a TB of data in mine, spread across 3k of files and not had a hiccup in years.
btsync : Use old version, there is no limit
Thank you! I may consider starting with owncloud alone and then switching sync client if I don't like it.
Sure, but no update doesn't sound like a good idea to me. Thanks for the suggestion, though.
Rsync for one way sync (works REALLY well even with tens of thousands of files) and unison apparently works for two way sync.
Yep, considered this myself, too. Rsync sure is a great tool and I love it for server to server sync. For personal use, I'd prefer real-time block-level sync, though. Rsync would definitely be my last resort. Thank you!
If one-way sync works,too then lsyncd might be worth looking at for real-time sync (idk what block-level sync means).
Sorry for not enough clarification - I want two-way sync.
By the way, lsyncd looks very interesting. I have a different data synced server to server by rsync+cron and I want to give lsyncd a try. Thanks for the suggestion again!
Have you given thought to what files are essential (those that require real-time sync)? I prefer to rsync non-essential files once a day, where as I'll real-time sync my current work folders (code, documents, and configs). Once a project is done, it's moved to an archive and rsync takes over.
The above approach may not suit your needs, but I've found it to be rock solid and light on resources (instead of polling/syncing files 24/7).
I see. Great point. I was just blindly dumping everything in Dropbox but now that I think about it, I might not need real-time sync of all of them. Thanks for pointing this out.
Putting all of the suggestions together, I'll move most of unused projects to an archive folder where I rsync and sync few working files using owncloud sync client.
More suggestions or your experience regarding Dropbox replacement appreciated.
@Jun I love those mangoes but they don't sell them here
Pydio maybe another option
I use Pydio with syncthing and haven't had an issue.
Same here.
For clarity, in my solution a) users only manage their own files so I don't have lots of users messing around with the same files and b) I use it with MySQL, not the SQLite default, which gets painfully slow past a very small amount of files.
If you're going to use btsync to get files into the own cloud directories, you may need to set up a regular job to run the 'update folders' .php file, as sometimes the web client fails to see changes not made through itself or the official sync client.
I sync a few thausand files with Owncloud. There were a few conflicts in early days but not a glitch in the last year or so. Its linux client is getting pretty nifty, v2.0 has multi account support and it's pretty fast in my experience.
I just dumped 30k files into owncloud last night, took them all in no bother :P
I like @Nekki also run it on MySQL.
May try seafile.. it has a good encryption too so that the host can not see inside the files.. should you worry about privacy.
Not self hosted, but have you considered Microsoft Onedrive for business? Perfect two way windows syncing without any problem. If you have multiple users that use the files you can utilize checking in/out with version history also.
It's great as long as the machine it's running on is stable if the machine it's on crashes during write it's possible to trash the library the repair tool is supposed to revert the repo back to the last stable state but it doesn't always work.
It also uploads in batches so the progress indicator on the sync client is largely meaningless.
Using owncloud at the moment because it stores the files natively on the filesystem (I.e even if owncloud goes splat I can still SSH in and access them)
I am using sync.com which is basically Dropbox but it doesn't have the encryption issues. They do not have your password in any way, shape or form. Also, they are based in Canada, not the US.
Pricing is reasonable, 500Gb is is enough for you at $49/yr, 2TB is on currently on special for $98/yr.
On the free accounts, starts at 5GB free, add a 1GB referral (and I get 1GB too), and there is an easy few steps for another free Gig so you get to 7GB fast. Get a few referrals and you are off.
The limit is not there anymore
Uhm, no it is not? At http://www.time4vps.eu/ you can get a 512GB VPS for 18 EUR/year (more than 2x cheaper); not to mention it's a whole VPS with 512 MB of RAM, so you can run other tasks on it as well.
Oh okay I see now.
The pricing is cheaper than like services. Dropbox is $99/yr for 1TB.
Not everyone wants to use a VPS with Owncloud etc. The free level is fine for me at the moment.
I'm upfront about that. You can use it or not. Simple.
Just synced 750gb in several thousand files over rsync.
mega.co.nz ... they give you free 50 gb