Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


RSYNC to remote server, issues
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

RSYNC to remote server, issues

libro22libro22 Member
edited July 2012 in General

Hello guys,

I rsync some files to a remote server weekly and the size is big (at least 50gb). When I do it via terminal, I do not get cryptic error messages but when it runs on cron, issues arise. Both IPs are on the allowed list of CSF.

Here are some of the messages:

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]

Read from remote host server@hostname: Connection reset by peer
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (58587 bytes received so far) [sender]

rsync: connection unexpectedly closed (224 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

I get these messages via e-mail within the first 40 mins and after a few hours. Any advise on this please?

I'm searching some solutions on Google as I write this, I hope you can provide your feedback too.

Thank you.

Comments

  • yomeroyomero Member
    edited July 2012

    You can use SCP without issues?
    The servers have enough disk space?
    Enough RAM?

  • vdnetvdnet Member

    What rsync version / command are you running?

    Try excluding -z (compression)

  • raindog308raindog308 Administrator, Veteran

    rsync with deep trees uses a lot of memory. Perhaps you should try running it, while running free -m every 30 seconds and capturing the output. Then look to see if memory is being exhausted.

    I've seen errors like yours when using small-memory VPSes to do rsync. Increasing RAM fixed the problem. It isn't so much the size of the file as it is the complexity of the directory structure. One 50GB file takes less memory for rsync than, say, /usr/src/linux.

    Thanked by 1marrco
  • @yomero, yes SCP is working, plenty of disk space. RAM on the remote server is quite small as its a LEB. I was not able to monitor the RAM on the remove server during the transfer.

    @vdnet rsync 3.0.6-4.el5_7.1 on both boxes, source is using 64bit while remote is using 32bit though. command: rsync -a -e "ssh -p XXX" /home/source/dir/ user@host:/home/remote/dir/

  • @raindog308 oh i see, i guess i have to check on that. thank you!

  • yomeroyomero Member

    How much is "quite small". If you are not able to monitor it because the lack of ram to start another ssh connection, then that is your problem.

  • @yomero about 128mb, I haven't actually tried to do that sorry, I will try it today and post some updates. thank you.

  • If it's OpenVZ, look at user_beancounters to see if you hit your limit. See failcnt column.

  • Having said that, if it's a memory issue, rsync will tell you out of memory on flist expand or some such messaage.

  • vdnetvdnet Member
    edited July 2012

    @libro22 Try: rsync -v -v -a -e "ssh -p XXX" /home/source/dir/ user@host:/home/remote/dir/ >debug.log 2>&1

    That should give you more details on the problem in debug.log. That being said, it could very well be a memory issue as others have said. The reason I mentioned the -z flag above is compression can require more memory.

Sign In or Register to comment.