New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Simulataneous remote sync with a bunch of LES VPS?
So I bought one of the LES bundles Anthony offered and is trying to set up a small SqlLite-backend HA blog using RoundRobin from Cloudflare and Nginx as the front-end.
Because the way it is set up, any comment could be posted on any one of the four VPS and therefore I am looking for a way to constantly keep the contents among the folder in sync. Bear in mind that these VPS are OpenVZ so no kernel module can be installed and also I hope it won't be too CPU or memory-heavy (I would assume a 5 minutes sync interval would be perfect).
Comments
sync all of server using unison (http://www.cis.upenn.edu/~bcpierce/unison/) much better than rsync, since rsync is only one way synchronization
Looks like an easy one...I'll give it a try. Thanks!
GlusterFS is what you're looking for, basically can make "RAID1" from folders across many servers, with guaranteed consistency (working file locking, etc), not every 5 minutes, but in real-time! But I am not sure how well it will work on low-RAM VMs.
MooseFS works too (for a distributed fault-tolerant FS like GlusterFS), and I've used it and it doesn't seem to use much memory (although currently not doing disk I/O). It also doesn't need FUSE or anything for the chunk servers.
@perennate there's a number of similar projects, also XtreemFS, Ceph... etc;
https://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems
it was my impression that GlusterFS had the combination of being still alive/maintained and the least awkward to set up and use. Particularly MooseFS put me off by the need of first install and configure three servers of different kind (metadata.. metalogger.. chunk server?... https://en.wikipedia.org/wiki/MooseFS ), just to get even something simple off the ground. Also with such design it seems to have a central point of failure (the metadata server), GlusterFS seemed to be more cleanly peer-to-peer.
I wish there was a good filesystem with an erasure coding storage backend. Tahoe-LAFS is neat but not intended for high-performance and doesn't store file attributes.
Can you install GlusterFS storage node on OpenVZ without FUSE?
Edit: I guess this doesn't apply here since anyway here the DFS would need to be mounted on VPS; but still wondering
Edit2: oh this seems interesting, hasn't been updated recently though: https://github.com/madiator/HadoopUSC
Probably not, but I assume that while you can't install your own modules on OpenVZ, with most providers (and probably at LowEndSpirit too) you can request something which is already included in the kernel and as popular as FUSE to be enabled/loaded for your VPS.
I'm using fuse on an ovz vps. was easy to install
Unfortunately I don't think LES has Fuse support
This has to be enabled on the provider's end, no?
I have a glusterfs "drive" on a few KVMs. It works great, but I can't mount glusterfs drive in openVZ container.
Edit: took out bad part ..
If LES doesn't allow you to request to enable FUSE (which is reasonable since the price is so low, they can't deal with all these kinds of requests) then you can try btsync (this is proprietary software), Unison, etc.
@perennate, @zhuanyi
I am a BTSync fanboy i use it for almost everything, i even made a DNS failover system using maradns and btsync - but be warned, btsync burns a lot cpu and memory.
...then cancel your LES, but I really doubt that is indeed the case.
What "all these kinds"? There are basically two popular requests (not even kinds), that people are likely to make: 1) TUN/TAP, 2) FUSE. And LES does already support the 1st one.
I recommend using inotify tools along with unison, it will enable faster propagation if used alongside ssh keys. I also believe its better at detecting changes inside a directory, for my project I started with unison but combined it with inotify tools for a better soloution
How is the inotify memory usage? Obviously with LES we have relatively limited memory available in the super-cheap VPS. And more importantly, does inotify support IPv6? I know Unison fortunately does.
@zhuanyi
Mine currently idles at 0.0 cpu and ram, and appears not to really use anything when unzipping a folder into a synced dir also, unison itself uses some resources when transferring but inotify-tools works as nicely as id expect. I'm going to be running a few of them for my project. its a geo replicated web server / multi-master MySQL cluster so id like things to be low usage and quick :P. But cant speak for your project just test it out and see how it works for you.
Also, inotify-tools just checks the changes in the directory, I use unison as well in the script, it was because when changes are made in subdirectories I seen that they weren't always triggering unison, also unison usually run with cronjob where as if you use inotify then you can be pretty much instant - transfer time.