71.1. rsync - fast remote file copy program (like rcp)
rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License version 2 and is currently being maintained by Wayne Davison.
71.1.1. 安装Rsync与配置守护进程
71.1.1.1. install with source
过程 71.1. rsync
- 安装rsync
在AS3 第二张CD上找到rsync-2.5.6-20.i386.rpm
[root@linuxas3 root]# cd /mnt [root@linuxas3 mnt]# mount cdrom [root@linuxas3 mnt]# cd cdrom/RedHat/RPMS [root@linuxas3 RPMS]# rpm -ivh rsync-2.5.6-20.i386.rpm
- 配置/etc/rsyncd.conf
在rh9,as3系统上rsync安装后,并没有创建rsyncd.conf文档,要自己创建rsyncd.conf文档
[root@linuxas3 root]# vi /etc/rsyncd.conf uid=nobody gid=nobody max connections=5 use chroot=no log file=/var/log/rsyncd.log pid file=/var/run/rsyncd.pid lock file=/var/run/rsyncd.lock #auth users=root secrets file=/etc/rsyncd.passwd [postfix] path=/var/mail comment = backup mail ignore errors read only = yes list = no auth users = postfix [netkiller] path=/home/netkiller/web comment = backup 9812.net ignore errors read only = yes list = no auth users = netkiller [pgsqldb] path=/var/lib/pgsql comment = backup postgresql database ignore errors read only = yes list = no
- 选项说明
uid = nobody gid = nobody use chroot = no # 不使用chroot max connections = 4 # 最大连接数为4 pid file = /var/run/rsyncd.pid #进程ID文件 lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log # 日志记录文件 secrets file = /etc/rsyncd.pwd # 认证文件名,主要保存用户密码,权限建议设为600,所有者root [module] # 这里是认证的模块名,在client端需要指定 path = /var/mail # 需要做镜像的目录 comment = backup xxxx # 注释 ignore errors # 可以忽略一些无关的IO错误 read only = yes # 只读 list = no # 不允许列文件 auth users = postfix # 认证的用户名,如果没有这行,则表明是匿名 [other] path = /path/to... comment = xxxxx
- 密码文件
在server端生成一个密码文件/etc/rsyncd.pwd
[root@linuxas3 root]# echo postfix:xxx >>/etc/rsyncd.pwd [root@linuxas3 root]# echo netkiller:xxx >>/etc/rsyncd.pwd [root@linuxas3 root]# chmod 600 /etc/rsyncd.pwd
- 启动rsync daemon
[root@linuxas3 root]# rsync --daemon
- 选项说明
- 添加到启动文件
echo "rsync --daemon" >> /etc/rc.d/rc.local [ OK ]
cat /etc/rc.d/rc.local 确认一下
- 测试
[root@linux docbook]# rsync rsync://netkiller.8800.org/netkiller [root@linux tmp]# rsync rsync://netkiller@netkiller.8800.org/netkiller Password: [chen@linux temp]$ rsync -vzrtopg --progress --delete postfix@netkiller.8800.org::postfix /tmp Password:
71.1.1.2. install with aptitude
过程 71.2. installation setp by setp
- installation
$ sudo apt-get install rsync
- enable
$ sudo vim /etc/default/rsync RSYNC_ENABLE=true
- config /etc/rsyncd.conf
$ sudo vim /etc/rsyncd.conf uid=nobody gid=nobody max connections=5 use chroot=no pid file=/var/run/rsyncd.pid lock file=/var/run/rsyncd.lock log file=/var/log/rsyncd.log #auth users=root secrets file=/etc/rsyncd.secrets [neo] path=/home/neo/www comment = backup neo ignore errors read only = yes list = no auth users = neo [netkiller] path=/home/netkiller/public_html comment = backup netkiller ignore errors read only = yes list = no auth users = netkiller [mirror] path=/var/www/netkiller.8800.org/html/ comment = mirror netkiller.8800.org exclude = .svn ignore errors read only = yes list = yes [music] path=/var/music comment = backup music database ignore errors read only = yes list = no [pgsqldb] path=/var/lib/pgsql comment = backup postgresql database ignore errors read only = yes list = no auth users = neo,netkiller
- /etc/rsyncd.secrets
$ sudo vim /etc/rsyncd.secrets neo:123456 netkiller:123456
$ sudo chmod 600 /etc/rsyncd.secrets
- start
$ sudo /etc/init.d/rsync start
- test
$ rsync -vzrtopg --progress --delete neo@localhost::neo /tmp/test1/ $ rsync -vzrtopg --progress --delete localhost::music /tmp/test2/
- firewall
$ sudo ufw allow rsync
71.1.1.3. xinetd
CentOS 6 之前的版本可以使用 xinetd, CentOS 7 不建议使用
yum install xinetd
配置 /etc/xinetd.d/rsync
vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable = yes flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID }
disable = yes 改为 disable = no
# vim /etc/rsyncd.conf
chkconfig xinetd on /etc/init.d/xinetd restart
71.1.1.4. CentOS 7 - systemctl
systemctl enable rsyncd systemctl start rsyncd systemctl restart rsyncd systemctl stop rsyncd
启动配置项 /etc/sysconfig/rsyncd
# cat /etc/sysconfig/rsyncd OPTIONS=""
启动脚本
# cat /usr/lib/systemd/system/rsyncd.service [Unit] Description=fast remote file copy program daemon ConditionPathExists=/etc/rsyncd.conf [Service] EnvironmentFile=/etc/sysconfig/rsyncd ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS" [Install] WantedBy=multi-user.target
71.1.2. rsyncd.conf
# Minimal configuration file for rsync daemon # See rsync(1) and rsyncd.conf(5) man pages for help # This line is required by the /etc/init.d/rsyncd script pid file = /var/run/rsyncd.pid port = 873 address = 192.168.1.171 #uid = nobody #gid = nobody uid = root gid = root use chroot = yes read only = yes #limit access to private LANs hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0 hosts deny=* max connections = 5 motd file = /etc/rsyncd/rsyncd.motd #This will give you a separate log file #log file = /var/log/rsync.log #This will log every file transferred - up to 85,000+ per user, per sync #transfer logging = yes log format = %t %a %m %f %b syslog facility = local3 timeout = 300 [home] path = /home list=yes ignore errors auth users = linux secrets file = /etc/rsyncd/rsyncd.secrets comment = linuxsir home exclude = beinan/ samba/ [beinan] path = /opt list=no ignore errors comment = optdir auth users = beinan secrets file = /etc/rsyncd/rsyncd.secrets [www] path = /www/ ignore errors read only = true list = false hosts allow = 172.16.1.1 hosts deny = 0.0.0.0/32 auth users = backup secrets file = /etc/backserver.pas [web_user1] path = /home/web_user1/ ignore errors read only = true list = false hosts allow = 202.99.11.121 hosts deny = 0.0.0.0/32 uid = web_user1 gid = web_user1 auth users = backup secrets file = /etc/backserver.pas [pub] comment = Random things available for download path = /path/to/my/public/share read only = yes list = yes uid = nobody gid = nobody auth users = pub secrets file = /etc/rsyncd.secrets
71.1.3. rsync 参数说明
命令行选项 -v, --verbose 详细模式输出 -q, --quiet 精简输出模式 -c, --checksum 打开校验开关,强制对文件传输进行校验 -a, --archive 归档模式,表示以递归方式传输文件,并保持所有文件属性,等于-rlptgoD -r, --recursive 对子目录以递归模式处理 -R, --relative 使用相对路径信息 -b, --backup 创建备份,也就是对于目的已经存在有同样的文件名时,将老的文件重新命名为~filename。可以使用--suffix选项来指定不同的备份文件前缀。 --backup-dir 将备份文件(如~filename)存放在在目录下。 -suffix=SUFFIX 定义备份文件前缀 -u, --update 仅仅进行更新,也就是跳过所有已经存在于DST,并且文件时间晚于要备份的文件。(不覆盖更新的文件) -l, --links 保留软链结 -L, --copy-links 想对待常规文件一样处理软链结 --copy-unsafe-links 仅仅拷贝指向SRC路径目录树以外的链结 --safe-links 忽略指向SRC路径目录树以外的链结 -H, --hard-links 保留硬链结 -p, --perms 保持文件权限 -o, --owner 保持文件属主信息 -g, --group 保持文件属组信息 -D, --devices 保持设备文件信息 -t, --times 保持文件时间信息 -S, --sparse 对稀疏文件进行特殊处理以节省DST的空间 -n, --dry-run现实哪些文件将被传输 -W, --whole-file 拷贝文件,不进行增量检测 -x, --one-file-system 不要跨越文件系统边界 -B, --block-size=SIZE 检验算法使用的块尺寸,默认是700字节 -e, --rsh=COMMAND 指定使用rsh、ssh方式进行数据同步 --rsync-path=PATH 指定远程服务器上的rsync命令所在路径信息 -C, --cvs-exclude 使用和CVS一样的方法自动忽略文件,用来排除那些不希望传输的文件 --existing 仅仅更新那些已经存在于DST的文件,而不备份那些新创建的文件 --delete 删除那些DST中SRC没有的文件 --delete-excluded 同样删除接收端那些被该选项指定排除的文件 --delete-after 传输结束以后再删除 --ignore-errors 及时出现IO错误也进行删除 --max-delete=NUM 最多删除NUM个文件 --partial 保留那些因故没有完全传输的文件,以是加快随后的再次传输 --force 强制删除目录,即使不为空 --numeric-ids 不将数字的用户和组ID匹配为用户名和组名 --timeout=TIME IP超时时间,单位为秒 -I, --ignore-times 不跳过那些有同样的时间和长度的文件 --size-only 当决定是否要备份文件时,仅仅察看文件大小而不考虑文件时间 --modify-window=NUM 决定文件是否时间相同时使用的时间戳窗口,默认为0 -T --temp-dir=DIR 在DIR中创建临时文件 --compare-dest=DIR 同样比较DIR中的文件来决定是否需要备份 -P 等同于 --partial --progress 显示备份过程 -z, --compress 对备份的文件在传输时进行压缩处理 --exclude=PATTERN 指定排除不需要传输的文件模式 --include=PATTERN 指定不排除而需要传输的文件模式 --exclude-from=FILE 排除FILE中指定模式的文件 --include-from=FILE 不排除FILE指定模式匹配的文件 --version 打印版本信息 --address 绑定到特定的地址 --config=FILE 指定其他的配置文件,不使用默认的rsyncd.conf文件 --port=PORT 指定其他的rsync服务端口 --blocking-io 对远程shell使用阻塞IO -stats 给出某些文件的传输状态 --progress 在传输时现实传输过程 --log-format=formAT 指定日志文件格式 --password-file=FILE 从FILE中得到密码 --bwlimit=KBPS 限制I/O带宽,KBytes per second -h, --help 显示帮助信息
71.1.3.1. -n, --dry-run perform a trial run with no changes made
模拟运行,显示日志,但不做复制操作。
rsync -anvzP /www/* root@172.16.0.1/www
71.1.3.2. --bwlimit=KBPS limit I/O bandwidth; KBytes per second
速度限制,限制为 100k Bytes/s
rsync -auvzP--bwlimit=100 /www/* root@172.16.0.1/www
71.1.3.3. -e, --rsh=COMMAND specify the remote shell to use
rsync -auzv --rsh=ssh root@202.130.101.33:/www/example.com/* /backup/example.com/ # --rsh=ssh 可以省略 rsync -auzv root@202.130.101.33:/www/example.com/* /backup/example.com/
如果需要特别参数,可以这样写,这里指定连接SSH的端口为20
rsync -auzv --rsh='ssh -p20' root@202.130.101.34:/www/example.com/* /backup/example.com/
71.1.4. step by step to learn rsync
- transfer file from src to dest directory
neo@netkiller:/tmp$ mkdir rsync neo@netkiller:/tmp$ cd rsync/ neo@netkiller:/tmp/rsync$ ls neo@netkiller:/tmp/rsync$ mkdir src dest neo@netkiller:/tmp/rsync$ echo file1 > src/file1 neo@netkiller:/tmp/rsync$ echo file2 > src/file2 neo@netkiller:/tmp/rsync$ echo file3 > src/file3
- skipping directory
neo@netkiller:/tmp/rsync$ mkdir src/dir1 neo@netkiller:/tmp/rsync$ mkdir src/dir2 neo@netkiller:/tmp/rsync$ rsync src/* dest/ skipping directory src/dir1 skipping directory src/dir2
- recurse into directories
neo@netkiller:/tmp/rsync$ rsync -r src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file2 file3
- backup
neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file1.2008-11-21 file2 file2.2008-11-21 file3 file3.2008-11-21 neo@netkiller:/tmp/rsync$
backup-dir
neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 --backup-dir mybackup src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file1.2008-11-21 file2 file2.2008-11-21 file3 file3.2008-11-21 mybackup neo@netkiller:/tmp/rsync$ ls dest/mybackup/ file1.2008-11-21 file2.2008-11-21 file3.2008-11-21
rsync -r --backup --suffix=.2008-11-21 --backup-dir ../mybackup src/* dest/ neo@netkiller:/tmp/rsync$ ls dest mybackup src neo@netkiller:/tmp/rsync$ ls src/ dir1 dir2 file1 file2 file3
- update
neo@netkiller:/tmp/rsync$ rm -rf dest/* neo@netkiller:/tmp/rsync$ rsync -r -u src/* dest/ neo@netkiller:/tmp/rsync$ echo netkiller>>src/file2 neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/ building file list ... done file2 sent 166 bytes received 42 bytes 416.00 bytes/sec total size is 38 speedup is 0.18
update by time and size
neo@netkiller:/tmp/rsync$ echo Hi>src/dir1/file1.1 neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/ building file list ... done dir1/file1.1 sent 166 bytes received 42 bytes 416.00 bytes/sec total size is 41 speedup is 0.20
- --archive
rsync -a src/ dest/
- --compress
rsync -a -z src/ dest/
- --delete
src
svn@netkiller:~$ ls src/ dir1 dir2 file1 file2 file3
dest
neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest svnroot@127.0.0.1's password: receiving file list ... done created directory /tmp/dest src/ src/file1 src/file2 src/file3 src/dir1/ src/dir2/ sent 104 bytes received 309 bytes 118.00 bytes/sec total size is 0 speedup is 0.00
src
svn@netkiller:~$ rm -rf src/file2 svn@netkiller:~$ rm -rf src/dir2
dest
neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest svnroot@127.0.0.1's password: receiving file list ... done deleting src/dir2/ deleting src/file2 src/ sent 26 bytes received 144 bytes 68.00 bytes/sec total size is 0 speedup is 0.00
71.1.5. rsync examples
http://samba.anu.edu.au/rsync/examples.html
71.1.5.1. upload
$ rsync -v -u -a --delete --rsh=ssh --stats localfile username@hostname:/home/username/
for example:
I want to copy local workspace of eclipse directory to another computer.
$ rsync -v -u -a --delete --rsh=ssh --stats workspace neo@192.168.245.131:/home/neo/
71.1.5.2. download
$ rsync -v -u -a --delete --rsh=ssh --stats neo@192.168.245.131:/home/neo/* /tmp/
71.1.5.3. mirror
rsync使用方法
rsync rsync://认证用户@主机/模块
rsync -vzrtopg --progress --delete 认证用户@主机::模块 /mirror目录
71.1.5.4. rsync delete
例 71.1. examples
用rsync删除目标目录
mkdir /root/blank
rsync --delete-before -a -H -v --progress --stats /root/blank/ ./cache/
71.1.5.5. backup to a central backup server with 7 day incremental
例 71.2. backup to a central backup server with 7 day incremental
#!/bin/sh # This script does personal backups to a rsync backup server. You will end up # with a 7 day rotating incremental backup. The incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tridge@linuxcare.com # directory to backup BDIR=/home/$USER # excludes file - this contains a wildcard pattern per line of files to exclude EXCLUDES=$HOME/cron/excludes # the name of the backup machine BSERVER=owl # your password on the backup server export RSYNC_PASSWORD=XXXXXX ######################################################################## BACKUPDIR=`date +%A` OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete --backup --backup-dir=/$BACKUPDIR -a" export PATH=$PATH:/bin:/usr/bin:/usr/local/bin # the following line clears the last weeks incremental directory [ -d $HOME/emptydir ] || mkdir $HOME/emptydir rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/ rmdir $HOME/emptydir # now the actual transfer rsync $OPTS $BDIR $BSERVER::$USER/current
71.1.5.6. backup to a spare disk
例 71.3. backup to a spare disk
I do local backups on several of my machines using rsync. I have an extra disk installed that can hold all the contents of the main disk. I then have a nightly cron job that backs up the main disk to the backup. This is the script I use on one of those machines. #!/bin/sh export PATH=/usr/local/bin:/usr/bin:/bin LIST="rootfs usr data data2" for d in $LIST; do mount /backup/$d rsync -ax --exclude fstab --delete /$d/ /backup/$d/ umount /backup/$d done DAY=`date "+%A"` rsync -a --delete /usr/local/apache /data2/backups/$DAY rsync -a --delete /data/solid /data2/backups/$DAY The first part does the backup on the spare disk. The second part backs up the critical parts to daily directories. I also backup the critical parts using a rsync over ssh to a remote machine.
71.1.5.7. mirroring vger CVS tree
例 71.4. mirroring vger CVS tree
The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via anonymous rsync using the following script. #!/bin/bash cd /var/www/cvs/vger/ PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin RUN=`lps x | grep rsync | grep -v grep | wc -l` if [ "$RUN" -gt 0 ]; then echo already running exit 1 fi rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog sum1=`sum $HOME/ChangeLog` sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog` if [ "$sum1" = "$sum2" ]; then echo nothing to do exit 0 fi rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/ exit 0 Note in particular the initial rsync of the ChangeLog to determine if anything has changed. This could be omitted but it would mean that the rsyncd on vger would have to build a complete listing of the cvs area at each run. As most of the time nothing will have changed I wanted to save the time on vger by only doing a full rsync if the ChangeLog has changed. This helped quite a lot because vger is low on memory and generally quite heavily loaded, so doing a listing on such a large tree every hour would have been excessive.
71.1.5.8. automated backup at home
例 71.5. automated backup at home
I use rsync to backup my wifes home directory across a modem link each night. The cron job looks like this #!/bin/sh cd ~susan { echo date dest=~/backup/`date +%A` mkdir $dest.new find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}" $dest.new \; cnt=`find $dest.new -type f | wc -l` if [ $cnt -gt 0 ]; then rm -rf $dest mv $dest.new $dest fi rm -rf $dest.new rsync -Cavze ssh . samba:backup } >> ~/backup/backup.log 2>&1 note that most of this script isn't anything to do with rsync, it just creates a daily backup of Susans work in a ~susan/backup/ directory so she can retrieve any version from the last week. The last line does the rsync of her directory across the modem link to the host samba. Note that I am using the -C option which allows me to add entries to .cvsignore for stuff that doesn't need to be backed up.
71.1.5.9. Fancy footwork with remote file lists
例 71.6. Fancy footwork with remote file lists
One little known feature of rsync is the fact that when run over a remote shell (such as rsh or ssh) you can give any shell command as the remote file list. The shell command is expanded by your remote shell before rsync is called. For example, see if you can work out what this does: rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/ note that that is backquotes enclosed by quotes (some browsers don't show that correctly).
71.1.6. rsync for windows
http://www.rsync.net/resources/howto/windows_rsync.html
71.1.7. 多进程 rsync 脚本
#!/usr/bin/perl my $path = "/data"; #本地目录 my $ip="172.16.xxx.xxx"; #远程目录 my $maxchild=5; #同时并发的个数 open FILE,"ls $path|"; while() { chomp; my $filename = $_; my $i = 1; while($i<=1){ my $un = `ps -ef |grep rsync|grep -v grep |grep avl|wc -l`; $i =$i+1; if( $un < $maxchild){ system("rsync -avl --size-only $path/$_ $ip:$path &") ; }else{ sleep 5; $i = 1; } } }
原文出处:Netkiller 系列 手札
本文作者:陈景峯
转载请与作者联系,同时请务必标明文章原始出处和作者信息及本声明。