Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


File system help please
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

File system help please

raza19raza19 Veteran
edited August 2014 in General

Need advice on these two matters

1) We are using multiple virtual machines(vms) spread over several different physical ones. We simply want to be able to mirror few folders on vms A,B & C. Rsyncing both ways i.e. from A to B and then from B to A is simple but does not work in our case because lets say a user is connected to machine A and has 3 files, if he deletes any file and then an automated script runs Rsync from B to A the file will again be replicated. There is indeed a workaround with arguments instructing Rsync to leave deleted files but it all leads to a spaghetti mix. Isn't there something simple for this scenario ? I looked at gluster but it requires partitioning and access to disk, since most of the vms are using openvz this isn't a possibility. Hence, please advise/suggest on any possible solutions.

2) I have a portal where each user has an account and a file repository that goes with it. Whenever any user uploads a file using a form, the php app hashes the file contents first and checks to see if the file already exists in the system (even if uploaded by a different user). If the file exists, no new file is stored, instead the system only saves a row with the details of the uploading user and links to the original file. This method eliminates file duplicates and saves space . But we are encountering a problem now; every user also gets an ftp account which can be utilized to download his repository files, since only unique files are stored we need a way to show to every user all files from his repository. Since this isn't a simple ftp folder it requires tinkering around.. We have seen symlinks and file mounts , both are not practicable because the former blocks access on permission grounds and latter requires thousands of files to be mounted individually at runtime... Please suggest any solution/idea, etc

Thanks a lot.

Comments

  • 1) glusterfs+kvm will have better chance.

    2) think about COW in btrfs or similar.

    Thanked by 1raza19
  • perennateperennate Member, Host Rep

    For the first one, there are a few distributed filesystems that don't require special modules (like how glusterfs requires NFS). For example, LizardFS (https://github.com/lizardfs/lizardfs) master and chunkserver can run and store their files in any directory; it also doesn't need much memory to run. You will of course need to enable FUSE module on your OpenVZ VMs, but this should be easy for you to do.

    Thanked by 1raza19
  • @perennate said:
    For the first one, there are a few distributed filesystems that don't require special modules (like how glusterfs requires NFS). For example, LizardFS (https://github.com/lizardfs/lizardfs) master and chunkserver can run and store their files in any directory; it also doesn't need much memory to run. You will of course need to enable FUSE module on your OpenVZ VMs, but this should be easy for you to do.

    I have spent the last few hours on the direction you gave & its going to work :) & Although you only suggested the solution for the first one, the same advice has also been helpful for the ftp thing. Thank you for being awesome :) I am still waiting for your storage plans since our last convo.

Sign In or Register to comment.