Rsync:

Rsync can duplicate data between two directories whether the data is collocated on the same computer or over the network between two computers. The directories can be on any accessible random access storage devices (hard drives, thumb drives, etc). The data source must be readable and the destination must be writable.

The benefit of Rsync is the ability to update the mirrored backup with computed changes from the data source with a minimal load to system resources. It is very data bandwidth and system intensive to perform an entire copy of a directory structure each time a backup is to be performed. While the initial backup is no faster than a copy, Rsync has the ability to load the system in a minimal fashion by transferring only changes when subsequent rsync updates are performed.

Description of the rsync algorithm

Rsync command format: rsync [options] source-path destination-path

• Back-up (push) your photo directories to a second drive:
  rsync -av ~/Pictures /mnt/drive2
  This creates a backup of your photos in /mnt/drive2/Pictures/
  Back-up to a USB thumb drive is similar: rsync -av ~/Pictures /media/KINGSTON
  When you add new photos, just re-execute this rsync command to backup the latest changes.
  Note: The drive name will be dependent on the manufacturer.

  [Potential Pitfall]: Do not include the source directory name in the destination.
  rsync -av ~/Pictures /mnt/drive2/Pictures
  This will result in /mnt/drive2/Pictures/Pictures/
  Note that rsync destinations acts just like the cp and rcp commands.
  Also note that rsync -av ~/Pictures/ /mnt/drive2/ has a different behavior from rsync -av ~/Pictures /mnt/drive2

• Back-up (push) two directory source to a single directory on the second drive:
  rsync -av ~/Pictures ~/Images /mnt/drive2
  This creates a backup of your photos from the two directories in /mnt/drive2/Pictures/

• Sync directories and if any were deleted in ~/Pictures, also delete it in /mnt/drive2/Pictures/:
  rsync -a --progress --delete ~/Pictures /mnt/drive2
  This creates a backup in /mnt/drive2/Pictures/

• Sync one specific file in a directory:
  rsync -a ~/Pictures/Group-photo-2001-A.jpg /mnt/drive2/Pictures

• Sync a group of specific files in a directory:
  rsync -a ~/Pictures/2001-*.jpg /mnt/drive2/Pictures
  This creates a backup in /mnt/drive2/Pictures/
  Note that when transferring files only, the directory name has to be provided in the destination path.

• Sync files and directories listed in a file:
  rsync -ar --files-from=Filelist.txt ~/Data /mnt/drive2
  This creates a backup in /mnt/drive2/Data/
  Directory paths are included if specified with a closing slash such as pathx/pathy/pathz/. Path names must be terminated with a "/" or "/."

• Back-up (push) your source code and compress to save space. Ignore object files:
  rsync -avz --delete --exclude='*.o' --exclude='*.so' ~/src /mnt/drive2
  This creates a backup in /mnt/drive2/src/ but does not transfer files with the ".o" and ".so" extensions.

• Back-up (push) your source code and ignore object and shared object code files:
  rsync -av --delete --filter='+ *.[ch]' --filter='- *.o' ~/src /mnt/drive2
  This transfers files with the extension ".c" and ".h" but does not transfer object files with the ".o" extensions.
  same as rsync.exe -av --exclude='*.o' --filter='+ *.c *.h' ~/src /mnt/drive2

• Back-up (push) your source code and ignore CM directories, object and shared object code files:
  rsync -artv --progress --delete --filter='+ *.[ch]' --filter='- *.o' --exclude=".svn" ~/src /mnt/drive2
  Note that --exclude overides the include filter --filter='+ *.[ch]' so that ".c" and ".h" files under .svn/ are not copied.

• rsync -ar dir1 dir2
  This will copy dir1 into dir2 to give dir2/dir1 thus dir1/file.txt gets copied to dir2/dir1/file.txt
• rsync -ar dir1/ dir2/
  This will copy the contents of dir1 into dir2 to give dir2/contents-of-dir1, eg dir1/file.txt gets copied to dir2/file.txt
  The following all achieve the same results:
  • rsync -ar dir1/ dir2/
  • rsync -ar dir1/* dir2
  • rsync -ar dir1/* dir2/

Note that rsync will be able to handle files with blanks in the file name or directory name as well as with dashes ("-") or underscores ("_").

1. connect client to a sever running rsync in daemon mode
2. connect client to a sever using a ssh shell

• Double colon refers to a connection to a host running the rsync daemon in the format hostname::module/path where the module name is identified by the configuration in /etc/rsyncd.conf. The double colon is equivalent to using the URL prefix rsync://
• Single colon refers to the use of a remote shell
• No colon then the directory is considered to be local to the system.

The Rsync server is often referred to as rsyncd or the rsync daemon. This is in fact the same rsync executable run with the command line argument "--daemon". This can be run stand-alone or using xinetd as is typically configured on most Linux distributions.

File: /etc/xinetd.d/rsync

Default: "disable = yes". Change to "disable = no"

service rsync { disable = no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID }

Start/Re-start xinetd: /etc/rc.d/init.d/xinetd restart

Typical Linux distributions do not pre-configure rsync for server use. Both Ubuntu and Red Hat based distributions require that one generates the configuration file "/etc/rsyncd.conf"

File: /etc/rsyncd.conf

log file = /var/log/rsyncd.log
hosts allow = 192.17.39.244, 192.17.39.60
hosts deny = *
list = true
uid = root
gid = root
read only = false

[Proj1]
path = /tmp/Proj1
comment = Project 1 rsync directory

[ProjX]
path = /var/ProjX
comment = Project X rsync directory

Client command to rsync to the server:

Pull: rsync -avr server-host-name::Proj1 /home/user1/Proj1/Data
(eg. update mobile laptop from server backup)

First configure ssh for "password-less" login:
Note that current Linux distributions use ssh version 2 and rsa.

[user1@myclient ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/user1/.ssh/id_rsa.
Your public key has been saved in /home/user1/.ssh/id_rsa.pub.
The key fingerprint is:
aa:1c:76:33:8a:9c:10:51:............

• Local client (private key): ~/.ssh/id_rsa
• Contents (one line) of file (public key): ~/.ssh/id_rsa.pub to be copied into file on server: ~/.ssh/authorized_keys

[user1@myclient ~]$ ls -l ~/.ssh/id_rsa
-rw-------. 1 user1 user1 1675 Sep  7 14:55 /home/user1/.ssh/id_rsa

[user1@myclient ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub user1@remote-server
user1@remote-server's password: 

rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/Data remote-server:/mnt/supersan/Proj1

rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/Data user1@remote-server:/mnt/supersan/Proj1

SSH options may be put in the file ~/.ssh/config

• Sync local directory with a remote directory (push):
  rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/data/ user1@remote-server:/mnt/supersan/Proj1/data/
  This will alter the remote directory to match the local.
  Note that this is not the same as:
  rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/data user1@remote-server:/mnt/supersan/Proj1/data
  which will result in a remote directory: /mnt/supersan/Proj1/data/data/
• Sync local directory with a remote directory (pull):
  rsync -avr --rsh=/usr/bin/ssh user1@remote-server:/home/user1/Proj1/data/ /mnt/supersan/Proj1/data/
  This will alter the local directory to match the remote.

For more examples with the discussions of the nuances of rsync, see the examples above for the "Single Host Operation".

Rsync to Amazon AWS/EC2:

rsync -artv --delete -e "ssh -i /home/user1/.ssh/instance1.pem" /home/user1/dir1/dir2/          ec2-user@ec2-xxx-xxx-xxx-xxx.us-yyyy-1.compute.amazonaws.com:/home/ec2-user/dir1/dir2/

crontab:

Note that rsync is often used with cron to perform a nightly rsync.

* 2 * * * rsync -avr server-host-name::Proj1/html /var/www > /var/log/rsync 2>&1

Multiple solutions exist for MS/Windows rsync clients but they all rely on Cygwin. There is no other port of rsync.

• Cygwin: Work in Cygwin shell environment
• DeltaCopy (client and server)
  Uses Rsync algorithm and protocol
• QtdSync: (client and server) MS/Windows version includes Cygwin rsync and ssh (and more)
  [download]

Cygwin must be installed for the purpose of this tutorial.
Be sure to install the Cygwin component "Net" + "rsync".

  C:\cygwin\bin\rsync.exe 
               \ssh.exe
               \sshpass.exe
               \cygcrypto-0.9.8.dll
               \cyggcc_s-1.dll
               \cygiconv-2.dll
               \cgypopt-0.dll
               \cygssp-0.dll
               \cygwin1.dll
               \cygz.dll

@set PATH=C:\cygwin\bin
@set LIB=C:\cygwin\bin
rsync.exe -avr /cygdrive/c/MISC/JUNK  remote-server::Proj1

[Potential Pitfall]: The following command: rsync.exe -avr c:MISC\JUNK remote-server::Proj1 will produce an error because rsync will interpret the "C:" as the remote host and not a drive letter. Drive letters are not POSIX compliant.

[Potential Pitfall]: The following commands:

cd c:\
rsync.exe -avr MISC\JUNK  remote-server::Proj1

For more on Cygwin POSIX paths: http://cygwin.com/cygwin-ug-net/using.html#using-pathnames

I chose QtdSync/cygwin as it provided a GUI and an isolated Cygwin rsync executable and its dependencies.

To list GUI command line arguments: C:\Program Files\QtdSync\QtdSync.exe --help

Includes rsync Cygwin components:

  C:\Program Files\QtdSync\bin\rsync.exe 
                              \ssh.exe
                              \sshpass.exe
                              \cygcrypto-0.9.8.dll
                              \cyggcc_s-1.dll
                              \cygiconv-2.dll
                              \cgypopt-0.dll
                              \cygssp-0.dll
                              \cygwin1.dll
                              \cygz.dll
                          \etc\fstab

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

@set PATH=C:\Program Files\QtdSync\bin
@set LIB=C:\Program Files\QtdSync\bin
rsync.exe -avr /cygdrive/c/MISC/JUNK  remote-server::Proj1

Rsync daemon configuration requires installing rsync as a MS/Windows service. Use the Cygwin command cygrunsrv command (installed under the Cygwin "Admin" category).

@set PATH=C:\cygwin\bin
@set LIB=C:\cygwin\bin
cygrunsrv -I rsyncd -e CYGWIN=nontsec --path /cygdrive/c/cygwin/bin/rsync.exe \ 
          -a "--daemon --config=/etc/rsyncd.conf --no-detach"

cygrunsrv -I rsyncd -e CYGWIN=nontsec --path /cygdrive/c/cygwin/bin/rsync.exe \
          -a "--daemon --config=/etc/rsyncd.conf --no-detach" -u Administrator -w super-secret-password

Note that Cygwin applications refer to configuration file referenced within the Cygwin environment. As viewed by the Cygwin rsync application, the file /etc/rsyncd.conf is actually C:\cygwin\etc\rsyncd.conf as viewed by the MS/Windows OS.

Where the option -e CYGWIN=nontsec turns off Microsoft Windows NT security permissions. On MS/Windows 7 use the following c:\cygwin\etc\fstab settings:

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

This is to avoid the following error:
rsync: failed to modify permissions on xxxxfilenamexxxx: Permission denied

File: C:\cygwin\etc\rsyncd.conf

log file = /var/log/rsyncd
hosts allow = 192.17.39.244, 192.17.39.60
hosts deny = *
list = true
use chroot = false
strict modes = false
read only = false
ignore nonreadable = yes
dont compress = *.gz *.tgz *.zip *.rpm *.deb *.iso *.bz2 *.jpg *.mpg *.mpeg

[Proj1]
path = /cygdrive/c/Proj1
comment = Project 1 rsync directory

Where the Cygwin log file /var/log/rsyncd is C:\cygwin\var\log\rsyncd as viewed by the MS/Windows OS.

The Cygwin rsync directory path /cygdrive/c/Proj1 is C:\Proj1\ as viewed by the MS/Windows OS.

[Potential Pitfall]: You must check to see if the service has started. If not select "Start the service". The Microsoft Windows firewall may not allow a system to attach to rsyncd on port 873. My experience is that the service is detected by the OS and added to the firewall privileges. I did not have to configure the firewall, registration was automatic. If the rsyncd service did not get registered with the MS/Windows firewall, one may have to specifically grant the permission to to the rsyncd service.

Select "Control panel" + Windows Firewall":

To remove rsyncd from the available list of services, execute the following command from a "cmd" terminal:
c:\cygwin\bin\cygrunsrv -R rsyncd

Using telnet to test an rsync connection:

One can use telnet to test connectivity with an Rsync server:

[bash]$ telnet 192.168.1.12  873
Trying 192.168.1.12...
Connected to 192.168.1.12.
Escape character is '^]'.
@RSYNCD: 30.0
^C
Connection closed by foreign host.

This is representative of a telnet connection to a misconfigured rsync server or one which has not yet been started (nothing for the client to connect to):

[bash]$ telnet 192.168.1.13  873
Trying 192.168.1.13...
 telnet: connect to address 192.168.1.13: Connection timed out

Rsync client responses:

[bash]$ rsync -avr 192.168.1.12::Proj1 /home/user1/Test
rsync: failed to connect to 192.1.1.12: Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(124) [receiver=3.0.6]

Rsync logs:

2011/09/08 21:13:10 [1032] rsyncd version 2.6.6 starting, listening on port 873
2011/09/08 21:17:10 [3156] rsync error: received SIGUSR1 or SIGINT (code 20)

2011/09/08 21:19:10 [2796] rsync on Proj1/ from 192.168.1.10
2011/09/08 21:19:10 [2796] sent 940 bytes  received 334 bytes  total size 97

• Samba Rsync home page
• Lsyncd - Unix/Linux only. Monitor drive for changes (inotify) and perform an rsync any time a change is made.

• rsync.net
• s3rsync.com

Books:

"Backup & Recovery: Inexpensive Backup Solutions for Open Systems" by W. Curtis Preston ISBN # 0596102461, O'Reily Media (Jan 2007) Covers backup utilities including tar, dump, cpio, ntbackup, ditto, rsync, AMANDA, Bacula, BackupPC, rdiff-backup, and rsnapshot.

   

Advertisements