Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Dropbox alternative with robust sync capacity for a large number of files
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Dropbox alternative with robust sync capacity for a large number of files

JunJun Member

Hi all,

I know this topic had been discussed several times on this forum, but they are too generic to address my problem, so I decided to start a new thread. As many others here, I want self-hosted solution to replace Dropbox, but strongly focused on sync capacity.

I want to sync about 400GB, mostly code, git, dataset, and photo - no video or large-sized files, which means huge number of files to index.

Now, I looked up for several threads in this forum and reviews online and found that non of the reviews are written based on my scale of sync need. Some reviews that claimed to test a lot of files were tested at around 20GB scale. Yet, I think I got a general sense of the performance.

Dropbox: expensive, nsa, but the best sync capacity.

Owncloud: may hiccup on big amount of data, slow sync client, great web client

Seafile: seems like it has good, fast sync capacity - can anybody share experience on syncing many files?

Syncthing: good sync capacity but no native support for Windows

pyd.io: not enough reviews

btsync: great sync capacity but closed-source and 10 folders limit

Considering my need, I'm leaning toward btsync + owncloud web interface + encfs. Sync capacity, reliability, and data integrity is more important than NSA to me.

Could somebody share experience with any of the open source solution with huge number of files to sync like what I need? Or any suggestions appreciated.

Comments

  • Regarding owncloud, I've got over a TB of data in mine, spread across 3k of files and not had a hiccup in years.

  • btsync : Use old version, there is no limit

  • Nekki said: Regarding owncloud, I've got over a TB of data in mine, spread across 3k of files and not had a hiccup in years.

    Thank you! I may consider starting with owncloud alone and then switching sync client if I don't like it.

    dragon1993 said: btsync : Use old version, there is no limit

    Sure, but no update doesn't sound like a good idea to me. Thanks for the suggestion, though.

  • Rsync for one way sync (works REALLY well even with tens of thousands of files) and unison apparently works for two way sync.

    Thanked by 1rm_
  • 4n0nx said: Rsync for one way sync (works REALLY well even with tens of thousands of files) and unison apparently works for two way sync.

    Yep, considered this myself, too. Rsync sure is a great tool and I love it for server to server sync. For personal use, I'd prefer real-time block-level sync, though. Rsync would definitely be my last resort. Thank you!

    Thanked by 14n0nx
  • Jun said: Yep, considered this myself, too. Rsync sure is a great tool and I love it for server to server sync. For personal use, I'd prefer real-time block-level sync, though. Rsync would definitely be my last resort. Thank you!

    If one-way sync works,too then lsyncd might be worth looking at for real-time sync (idk what block-level sync means). :)

  • 4n0nx said: If one-way sync works,too then lsyncd might be worth looking at for real-time sync (idk what block-level sync means). :)

    Sorry for not enough clarification - I want two-way sync.

    By the way, lsyncd looks very interesting. I have a different data synced server to server by rsync+cron and I want to give lsyncd a try. Thanks for the suggestion again!

    Thanked by 14n0nx
  • @Jun said:
    Yep, considered this myself, too. Rsync sure is a great tool and I love it for server to server sync. For personal use, I'd prefer real-time block-level sync, though. Rsync would definitely be my last resort. Thank you!

    Have you given thought to what files are essential (those that require real-time sync)? I prefer to rsync non-essential files once a day, where as I'll real-time sync my current work folders (code, documents, and configs). Once a project is done, it's moved to an archive and rsync takes over.

    The above approach may not suit your needs, but I've found it to be rock solid and light on resources (instead of polling/syncing files 24/7).

  • telephone said: Have you given thought to what files are essential (those that require real-time sync)? I prefer to rsync non-essential files once a day, where as I'll real-time sync my current work folders (code, documents, and configs). Once a project is done, it's moved to an archive and rsync takes over.

    I see. Great point. I was just blindly dumping everything in Dropbox but now that I think about it, I might not need real-time sync of all of them. Thanks for pointing this out.

    Putting all of the suggestions together, I'll move most of unused projects to an archive folder where I rsync and sync few working files using owncloud sync client.

    More suggestions or your experience regarding Dropbox replacement appreciated.

    Thanked by 1telephone
  • @Jun I love those mangoes but they don't sell them here :(

  • Pydio maybe another option

  • I use Pydio with syncthing and haven't had an issue.

  • @Nekki said:
    Regarding owncloud, I've got over a TB of data in mine, spread across 3k of files and not had a hiccup in years.

    Same here.

  • Jun said: Thank you! I may consider starting with owncloud alone and then switching sync client if I don't like it.

    For clarity, in my solution a) users only manage their own files so I don't have lots of users messing around with the same files and b) I use it with MySQL, not the SQLite default, which gets painfully slow past a very small amount of files.

    If you're going to use btsync to get files into the own cloud directories, you may need to set up a regular job to run the 'update folders' .php file, as sometimes the web client fails to see changes not made through itself or the official sync client.

  • I sync a few thausand files with Owncloud. There were a few conflicts in early days but not a glitch in the last year or so. Its linux client is getting pretty nifty, v2.0 has multi account support and it's pretty fast in my experience.

  • I just dumped 30k files into owncloud last night, took them all in no bother :P

    I like @Nekki also run it on MySQL.

  • May try seafile.. it has a good encryption too so that the host can not see inside the files.. should you worry about privacy.

  • StealthyHostingStealthyHosting Member, Host Rep
    edited September 2015

    Not self hosted, but have you considered Microsoft Onedrive for business? Perfect two way windows syncing without any problem. If you have multiple users that use the files you can utilize checking in/out with version history also.

  • @Wira_Soenaryo said:
    May try seafile.. it has a good encryption too so that the host can not see inside the files.. should you worry about privacy.

    It's great as long as the machine it's running on is stable if the machine it's on crashes during write it's possible to trash the library the repair tool is supposed to revert the repo back to the last stable state but it doesn't always work.

    It also uploads in batches so the progress indicator on the sync client is largely meaningless.

    Using owncloud at the moment because it stores the files natively on the filesystem (I.e even if owncloud goes splat I can still SSH in and access them)

  • Monsta_AUMonsta_AU Member
    edited September 2015

    Jun said: I want to sync about 400GB, mostly code, git, dataset, and photo - no video or large-sized files, which means huge number of files to index.

    I am using sync.com which is basically Dropbox but it doesn't have the encryption issues. They do not have your password in any way, shape or form. Also, they are based in Canada, not the US.

    Pricing is reasonable, 500Gb is is enough for you at $49/yr, 2TB is on currently on special for $98/yr.

    On the free accounts, starts at 5GB free, add a 1GB referral (and I get 1GB too), and there is an easy few steps for another free Gig so you get to 7GB fast. Get a few referrals and you are off.

  • NyrNyr Community Contributor, Veteran

    @Jun said:
    btsync: great sync capacity but closed-source and 10 folders limit

    The limit is not there anymore :)

  • rm_rm_ IPv6 Advocate, Veteran
    edited September 2015

    Monsta_AU said: Pricing is reasonable, 500Gb is is enough for you at $49/yr, 2TB is on currently on special for $98/yr.

    Uhm, no it is not? At http://www.time4vps.eu/ you can get a 512GB VPS for 18 EUR/year (more than 2x cheaper); not to mention it's a whole VPS with 512 MB of RAM, so you can run other tasks on it as well.

    Monsta_AU said: add a 1GB [referral]

    Oh okay I see now.

  • Monsta_AUMonsta_AU Member
    edited September 2015

    rm_ said: Uhm, no it is not? At http://www.time4vps.eu/ you can get a 512GB VPS for 18 EUR/year (more than 2x cheaper); not to mention it's a whole VPS with 512 MB of RAM, so you can run other tasks on it as well.

    The pricing is cheaper than like services. Dropbox is $99/yr for 1TB.

    Not everyone wants to use a VPS with Owncloud etc. The free level is fine for me at the moment.

    rm_ said: Oh okay I see now.

    I'm upfront about that. You can use it or not. Simple.

  • Just synced 750gb in several thousand files over rsync. :)

  • mega.co.nz ... they give you free 50 gb

Sign In or Register to comment.