Rsync out of memory? Try this...

pubcrawler

If you do proper backups and are a shell type person, you probably are an rsync user.

In the lowend market, where there is limited RAM available, rsync can really fail in a big way. Failure is out of memory related when rsync attempts to load all the filenames and info in to RAM at startup.

If you are getting out of memory errors and done the search engine researching and swear rsync has a bug, fear not.

This is your new rsync commandline option to postpend to your current script or cron job:

--no-inc-recursive

That cured our ills on a 256MB VPS. Bound to work just as well on smaller instances.

pubcrawler

I run into this issue all the time on smaller RAM ARM boxes. Haven't given it a spin yet on those. The feature was released somewhere in the 3.xxx release timeline, so it's "recent" and doesn't appear in the ancient write ups folks have on rsync from 2005

RobertClarke

Nice tip, will be using that on my new storage box, thanks.

Andre

Working well on my RPi, big improvement actually.

corpus

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter
--no-i-r alias.

Nice tip

pubcrawler

Glad it helps @Andre!

Thanks @corpus. I like your tag line there! Too bad apt-get won't a beer

Andre

@pubcrawler said: Too bad apt-get won't a beer

It won't if you're not running the command as superuser. (-:

Relevant xkcd

eastonch

@andre I just had to justify what sudo meant in class... Because I was in piss fits at that.

sleddog

You can also use something like,

--rsync-path="nice -n 19 /usr/bin/rsync"

To lower the priority of the resync process on the remote end. Apparently, with a modern kernel nice also adjusts IO priority.

Of course you'd also nice the local end with,

nice -n 19 /usr/bin/rsync ...

bnmkl

Thanks @pubcrawler, @sleddog. Very nice. :-)

@Andre - lol