Continuous synchronize between 2 or more servers

howardsl2

@joodle said:

SSHFS works but the files are stored only on the "master" server. All other servers only have a cached copy of recently accessed files. From what I understand, a round-trip delay to the "master" server would be required for first-time access to a file.

For a SSHFS tutorial on how to add "automatic reconnect" via autossh, and/or steps to harden the security settings, check out the related blog article in my signature.

BlazeMuis

Trying out pkr5770 thingy, looks to be faster/better than SSHFS

Thanks!

BotoX

I've been using lsync and it works pretty nice.

Edit source and targetlist, make an SSH key and a new user for it...
https://p.botox.bz/ituz.lua

darkshire

+1 to btsync

Master_Bo

I use GlusterFS to keep data in sync between 3+ servers (if you're running a cluster, I found a good idea to keep a 'file sync only' node that is never powered off (well, as never as possible). for better results.

To keep database in sync, do not try to sync their files via third-party tool like GlusterFS. The result will be corrupt databases on all nodes.

Use something like Galera library instead. Note: I use Galera version of MariaDB to keep databases in sync across nodes.

It's somewhat tricky and puts a number of restrictions on databases, but it all is well dcumented on MariaDB site anyway.

vfuse

If these are HTTP downloads just use nginx reverse proxy with proxy store, probably the easiest way.

pkr5770

@joodle

let me know how it goes

vdnet

joodle said: generated

@jvnadr said:
But is rsync able to synchronize in real time, including the databases? Could the db's be corrupted if being synchronized via rsync? Shouldn't sync with dumb or via other methods? Or is there a way to include database structures with rsync and they won't be corrupted to the mirror servers?

You shouldn't use rsync for database or commonly modified files. You should use block level replication. However, for the OPs use, rsync would work fine.

SplitIce

If you can settle for a minute or two delay bittorrent sync could be used. However synchronisation may not be what you need.

GlusterFS isnt synchronisation its distribution + replication. It is a very useful system when setup correctly. We presently use it for storing our web data (replication) and providing a shared filesytem internally at DigitalOcean.

I would recommend looking at distributed filesystems, as they sound more suitable for your application.

netomx

What do you recommend for a Owncloud mirror?

MuZo

@netomx if your nodes are in the same DC/network use GlusterFS. Keep in mind that the network speed between the 2 or more nodes will be your I/O speed of the distributed folder. A better solution when your nodes don't have such network speed between each other would be softwares that leave a local copy and sync when there is a difference.

I've tried doing a owncloud "cluster" with vpses by backupsy NL and prometeus but I could "only" reach around 13MB/s I/O and the owncloud website needed around 1-2min to load one page.
(my old thread: http://lowendtalk.com/discussion/23941/alternatives-to-gluster)