Looking for a good rsync command to backup disks and stuff

valk

I'm asking if there's any good rsync command that you guys use to backup files and parts to a remote server, preserve perms anything is fine. I'm currently using -avuzPh to rsync, and I dont think thats enough

If you have a better backup option other than rsync, tell me below

aj_potc

My go-to for general file transfer over a remote link has always been:

rsync -avz --progress

I know there are plenty more options, but that hasn't failed me yet.

Falzo

since you asked...

rsync -aAHe 'ssh -p1234 -T -c aes128-ctr -o Compression=no -x' --rsync-path="ionice nice rsync" --delete --numeric-ids --stats --exclude-from='/path/exclude.list' source.server:/ /path/to/rsync/copy

explained:
-aAH essentials includes all the fancy options for recursive, attributes and hardlinks, which can be listed seperately as well, if you feel the need to.
-e is just the needed flag to set custom ssh option, like a different port and f.i. disable compression. the latter buys in on better network speed and less CPU usage by the cost of higher traffic - as it's usually incremental I don't care about traffic, so...
--rsync-path - specify in which way to run rsync on the source system, I tend to use ionice/nice here to be ressourcefriendly while taking the backup
--delete removes files that got deleted on the source already
--numeric-ids keeps the original user/group-id numbers. this is useful if you want to be able to restore properly to the same system even if user were deleted etc.
--stats puts out some info after rsync finishes
--exclude-from='file' helps to exculde unneeded stuf from the beginning, like /dev /proc /sys or maybe /var/log/ or session dirs etc.

however there probably are a million ways to combine and use, so take every recommendation with a grain of salt, read through ma pages of rsync and find your own best combination ;-)

eva2000

For speed, better to just use newer rsync 3.2 with zstd and xxhash checksum support - way faster than rsync 3.1 from my benchmarks with rsync 3.2 with zstd capable of pushing 100-200+ MB/s transfer speed

darkimmortal

Don't forget the 'a' flag does not backup everything - consider the 'A' (ACLs), 'H' (hard links) and 'X' (xattrs) flags too depending on your use case, though in fairness these are niche scenarios

For backups I'm a fan of Borg

NetDynamics24

Also you can do :
rsync -rav

valk

@Falzo said:
since you asked...

rsync -aAHe 'ssh -p1234 -T -c aes128-ctr -o Compression=no -x' --rsync-path="ionice nice rsync" --delete --numeric-ids --stats --exclude-from='/path/exclude.list' source.server:/ /path/to/rsync/copy

explained:
-aAH essentials includes all the fancy options for recursive, attributes and hardlinks, which can be listed seperately as well, if you feel the need to.
-e is just the needed flag to set custom ssh option, like a different port and f.i. disable compression. the latter buys in on better network speed and less CPU usage by the cost of higher traffic - as it's usually incremental I don't care about traffic, so...
--rsync-path - specify in which way to run rsync on the source system, I tend to use ionice/nice here to be ressourcefriendly while taking the backup
--delete removes files that got deleted on the source already
--numeric-ids keeps the original user/group-id numbers. this is useful if you want to be able to restore properly to the same system even if user were deleted etc.
--stats puts out some info after rsync finishes
--exclude-from='file' helps to exculde unneeded stuf from the beginning, like /dev /proc /sys or maybe /var/log/ or session dirs etc.

however there probably are a million ways to combine and use, so take every recommendation with a grain of salt, read through ma pages of rsync and find your own best combination ;-)

Definitely gonna try this command once my rsync session's done. Thanks

coolgoole

@Falzo said:
since you asked...

rsync -aAHe 'ssh -p1234 -T -c aes128-ctr -o Compression=no -x' --rsync-path="ionice nice rsync" --delete --numeric-ids --stats --exclude-from='/path/exclude.list' source.server:/ /path/to/rsync/copy

explained:
-aAH essentials includes all the fancy options for recursive, attributes and hardlinks, which can be listed seperately as well, if you feel the need to.
-e is just the needed flag to set custom ssh option, like a different port and f.i. disable compression. the latter buys in on better network speed and less CPU usage by the cost of higher traffic - as it's usually incremental I don't care about traffic, so...
--rsync-path - specify in which way to run rsync on the source system, I tend to use ionice/nice here to be ressourcefriendly while taking the backup
--delete removes files that got deleted on the source already
--numeric-ids keeps the original user/group-id numbers. this is useful if you want to be able to restore properly to the same system even if user were deleted etc.
--stats puts out some info after rsync finishes
--exclude-from='file' helps to exculde unneeded stuf from the beginning, like /dev /proc /sys or maybe /var/log/ or session dirs etc.

however there probably are a million ways to combine and use, so take every recommendation with a grain of salt, read through ma pages of rsync and find your own best combination ;-)

Please forgive my bad English
My rsync cannot synchronize files under other usernames
My redis username is redis
My rsync username is root

#bin/bash
#cd
cd ~/.rsync_backup
#home
rsync -ar  ~/.tmux.conf ~/.rsync_backup
rsync -ar  ~/.zshrc ~/.rsync_backup
rsync -ar  ~/.vimrc ~/.rsync_backup
rsync -ar --delete ~/.vim ~/.rsync_backup
rsync -ar  ~/.config/rclone/rclone.conf ~/.rsync_backup
#rsync -ar  ~/mkdir-nginx-cache.sh ~/.rsync_backup
#ssh
sudo rsync -arR  /etc/ssh/sshd_config ~/.rsync_backup
#nginx
sudo rsync -arR --delete /etc/nginx/conf.d ~/.rsync_backup
sudo rsync -arR --delete /etc/nginx/global ~/.rsync_backup
sudo rsync -arR  /etc/nginx/nginx.conf ~/.rsync_backup
#mairadb
sudo rsync -arR  /etc/mysql/mariadb.cnf ~/.rsync_backup
#php
sudo rsync -arR  /etc/php/7.4/fpm/php-fpm.conf ~/.rsync_backup
sudo rsync -arR  /etc/php/7.4/fpm/pool.d/www.conf ~/.rsync_backup
sudo rsync -arR  /etc/php/7.4/fpm/php.ini ~/.rsync_backup
#fail2ban
sudo rsync -arR   /etc/fail2ban/jail.local ~/.rsync_backup
sudo rsync -arR   /etc/fail2ban/jail.d/jail-debian.local ~/.rsync_backup
#redis
sudo rsync -arR  /etc/redis/redis.conf ~/.rsync_backup
#rsync
sudo rsync -arR  /etc/rsyncd.conf ~/.rsync_backup
#letsencrypt
#sudo rsync -arR --delete /etc/letsencrypt ~/.rsync_backup
sudo tar -zcPf ~/.rsync_backup/letsencrypt.tar.gz /etc/letsencrypt

rsync cannot synchronize redis

sudo rsync -arR /etc/redis/redis.conf ~/.rsync_backup

rsync cannot synchronize letsencrypt

#sudo rsync -arR --delete /etc/letsencrypt ~/.rsync_backup
sudo tar -zcPf ~/.rsync_backup/letsencrypt.tar.gz /etc/letsencrypt

valk

@coolgoole said:
Please forgive my bad English
My rsync cannot synchronize files under other usernames
My redis username is redis
My rsync username is root

#bin/bash
#cd
cd ~/.rsync_backup
#home
rsync -ar  ~/.tmux.conf ~/.rsync_backup
rsync -ar  ~/.zshrc ~/.rsync_backup
rsync -ar  ~/.vimrc ~/.rsync_backup
rsync -ar --delete ~/.vim ~/.rsync_backup
rsync -ar  ~/.config/rclone/rclone.conf ~/.rsync_backup
#rsync -ar  ~/mkdir-nginx-cache.sh ~/.rsync_backup
#ssh
sudo rsync -arR  /etc/ssh/sshd_config ~/.rsync_backup
#nginx
sudo rsync -arR --delete /etc/nginx/conf.d ~/.rsync_backup
sudo rsync -arR --delete /etc/nginx/global ~/.rsync_backup
sudo rsync -arR  /etc/nginx/nginx.conf ~/.rsync_backup
#mairadb
sudo rsync -arR  /etc/mysql/mariadb.cnf ~/.rsync_backup
#php
sudo rsync -arR  /etc/php/7.4/fpm/php-fpm.conf ~/.rsync_backup
sudo rsync -arR  /etc/php/7.4/fpm/pool.d/www.conf ~/.rsync_backup
sudo rsync -arR  /etc/php/7.4/fpm/php.ini ~/.rsync_backup
#fail2ban
sudo rsync -arR   /etc/fail2ban/jail.local ~/.rsync_backup
sudo rsync -arR   /etc/fail2ban/jail.d/jail-debian.local ~/.rsync_backup
#redis
sudo rsync -arR  /etc/redis/redis.conf ~/.rsync_backup
#rsync
sudo rsync -arR  /etc/rsyncd.conf ~/.rsync_backup
#letsencrypt
#sudo rsync -arR --delete /etc/letsencrypt ~/.rsync_backup
sudo tar -zcPf ~/.rsync_backup/letsencrypt.tar.gz /etc/letsencrypt

rsync cannot synchronize redis

sudo rsync -arR /etc/redis/redis.conf ~/.rsync_backup

rsync cannot synchronize letsencrypt

#sudo rsync -arR --delete /etc/letsencrypt ~/.rsync_backup
sudo tar -zcPf ~/.rsync_backup/letsencrypt.tar.gz /etc/letsencrypt

Can you paste the error here so I can have a look at the problem

cpsd

i am always too lazy for this, that is why i prefer grsync (GUI at backup server).

its420somewhere

rsync -Aax

coolgoole

@valk said:

@coolgoole said:
Please forgive my bad English
My rsync cannot synchronize files under other usernames
My redis username is redis
My rsync username is root

redis --I found out what the problem is

-rw-r----- 1 redis redis 9.2K Aug 20 10:51 /etc/redis/redis.conf
Here is "640", not "644"

drwxr-xr-x 4 root root 4.0K Dec 12 00:10 accounts
drwx------ 5 root root 4.0K Dec 12 00:19 archive---Same here
-rw-r--r-- 1 root root  121 May 26  2018 cli.ini
drwxr-xr-x 2 root root 4.0K Dec 12 00:19 csr
drwx------ 2 root root 4.0K Dec 12 00:19 keys-----Same here
drwx------ 5 root root 4.0K Dec 12 00:19 live-----Same here
drwxr-xr-x 2 root root 4.0K Dec 12 00:19 renewal
drwxr-xr-x 5 root root 4.0K Dec 10 17:48 renewal-hooks

Same here.
Here is "700", not "755"

I do not know how to solve this

Falzo

@coolgoole said:

@valk said:

@coolgoole said:
Please forgive my bad English
My rsync cannot synchronize files under other usernames
My redis username is redis
My rsync username is root

redis --I found out what the problem is

-rw-r----- 1 redis redis 9.2K Aug 20 10:51 /etc/redis/redis.conf
Here is "640", not "644"

drwxr-xr-x 4 root root 4.0K Dec 12 00:10 accounts
drwx------ 5 root root 4.0K Dec 12 00:19 archive---Same here
-rw-r--r-- 1 root root  121 May 26  2018 cli.ini
drwxr-xr-x 2 root root 4.0K Dec 12 00:19 csr
drwx------ 2 root root 4.0K Dec 12 00:19 keys-----Same here
drwx------ 5 root root 4.0K Dec 12 00:19 live-----Same here
drwxr-xr-x 2 root root 4.0K Dec 12 00:19 renewal
drwxr-xr-x 5 root root 4.0K Dec 10 17:48 renewal-hooks

Same here.
Here is "700", not "755"

I do not know how to solve this

first of all: learn english. this is no criticism but unavoidable if you want to be able to mange your servers properly. otherwise you'll never understand properly what might be wrong or what people want to explain to you. there is no way around it.

second: DO NOT mess around with the permission on those files and folders!!
they are not world-readable for a reason.

most likely what you are trying to do there with running rsync in that script simply does not work like you think it does... especially for the sudo part. check the process list while it's running, but I doubt your rsync really runs as root there. it's rather still an unprivileged user, who then simply can't access these files (and right so).

also it does not make sense to run rsync seperately for each source dir or file. instead compile an include list of sources and use that with a single line of rsync.

coolgoole

@Falzo said:

@coolgoole said:

@valk said:

Thank you, I am learning English now, but it can't be done in a short time
You misunderstood, I didn't mean there was a problem with your answer
I'm looking at your answer, I happened to meet my problem, so I asked you
I encountered this problem a few days ago and did not solve it

There is no problem syncing on one vps
What I encountered was that the server and client could not be synchronized successfully

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1650) [generator=3.1.2]
Cause
Insufficient permissions on some files inside the subscription directory.

Daniel15

@aj_potc said:
My go-to for general file transfer over a remote link has always been:

rsync -avz --progress

I know there are plenty more options, but that hasn't failed me yet.

Would not recommend compression (z flag) unless you know it'll help (eg a slow connection with lots of compressable files, such as plain text or HTML files). Many times it just adds extra overhead, particularly if the connection between the two servers is very fast.

aj_potc

@Daniel15 said:

@aj_potc said:
My go-to for general file transfer over a remote link has always been:

rsync -avz --progress

I know there are plenty more options, but that hasn't failed me yet.

Would not recommend compression (z flag) unless you know it'll help (eg a slow connection with lots of compressable files, such as plain text or HTML files). Many times it just adds extra overhead, particularly if the connection between the two servers is very fast.

That's true, and I back up things such as databases and log files, where compression really helps.

I've done a little experimenting with this flag in the past, and unless the system is quite slow to begin with, I don't notice too much extra overhead.

Most of the overhead from rsync comes from its ravenous IO needs, and for this reason I like to prefix the command with ionice, as @Falzo mentioned already.