Continuous synchronize between 2 or more servers

BlazeMuis

I'm currently trying to setup a small cdn for the mods and custom maps download for my gameservers, so that users will get the best download speed possible

GeoDNS is being done via Rage4

But i just can't find out how to duplicate/sync all the files between the 2 servers (so that the files only have to be uploaded to 1 servers)

Would be great if someone could help me out

J1021

Rsync on cron?

matteob

@joodle said:

You can setup an DRDB enviroment

rm_

GlusterFS.

msg7086

https://github.com/hollow/inosync

Not tested yet but worth a try.

tomle

btsync is easy to get started with

ihatetonyy

rm_ said: GlusterFS.

rmlhhd

RSync

RyanD

Ceph or gluster.

Gluster is probably the simplest/quickest

bobby

apt-get install lsyncd

mikho

Unison or rsync

Profforg

Wow, LET so cool Never know about inosync and that GlusterFS is good for continous synchronize of files

BlazeMuis

Going to use Rsync

Anybody got any idea how to run it in the background so that on a change the file gets transfered?

J1021

@tomle said:
btsync is easy to get started with

The software becomes very unstable with lots of files (200/300k+) and begins to crash, not notice changes, spend hours re-indexing directories killing the CPU etc.

jvnadr

But is rsync able to synchronize in real time, including the databases? Could the db's be corrupted if being synchronized via rsync? Shouldn't sync with dumb or via other methods? Or is there a way to include database structures with rsync and they won't be corrupted to the mirror servers?

BlazeMuis

Could someone please answer my question? I generated a ssh key, put it on the remote host but still that retarded rsync asks my for a password

xDutchy

Databases can be synced with master-slave replication, prevents corruption etc

J1021

joodle said: Could someone please answer my question? I generated a ssh key, put it on the remote host but still that retarded rsync asks my for a password

Have you check /etc/ssh/sshd_config to ensure keys are enabled, make sure the public and private key are stored in the correct location? It should just work like a regular SSH session would.

BlazeMuis

1e10 said: Have you check /etc/ssh/sshd_config to ensure keys are enabled, make sure the public and private key are stored in the correct location? It should just work like a regular SSH session would.

Still asking for a damn password

globalRegisters

inotify/inotifywait and rsync

BlazeMuis

Got it working

Did the following:

ssh-keygen

Press Enter key till you get the prompt

ssh-copy-id -i root@ip_address

(It will once ask for the password of the host system)

ssh root@ip_address

Now you should be able to login without any password

taronyu

Isn't that how it works? What did you do before?

BlazeMuis

@taronyu said:
Isn't that how it works? What did you do before?

Followed this tutorial but didn't work: http://www.cyberciti.biz/tips/ssh-public-key-based-authentication-how-to.html

This is the first time i work with ssh keys though

Caveman122

You need to do the following:

create a config file in host1 for host1_user with the following name and content

~/.ssh/config

```
Host host2_alias
HostName host2.com
User host2_user
IdentityFile ~/.ssh/host2.com

~/.ssh/host2.com

>```
>The private key for host2 user host2_user
>```

Now when you enter

ssh host2_alias
```

It will use the key. The ~/.ssh/config and ~/.ssh/host2.com file needs to be 600

pkr5770

As suggested above use inotify + rsync good for checking recursive dirs. Now you have keys create a script to detect changes in a dir and run the command to rsync. I have successfully got this working bi directionaly although it seems you dont require this. If you need to do a database too use multi master mysql clusters with replication. If you want the script sync pm me i'll be on later

pkr5770

sorry, inotify /inotifywait + unison, its better for recursive detection than rsync in my experience.

pkr5770

save this script and sync.sh

http://pastebin.com/fjjZxbqe

start a screen session:
screen -a -m -d sync

enter screen
screen -x sync

start script with directory to monitor:
sh sync.sh /dir/on/local/machine

BlazeMuis

I'm currently using SSHFS, and it's working really good

pkr5770

Does't that mean the files aren't stored on the remote server? or are the files cached?

BlazeMuis

@pkr5770 said:
Does't that mean the files aren't stored on the remote server? or are the files cached?

It downloads the files from the remote/main server, works great

pkr5770

nice I'll check it out next time