Rsync: backup and restore:

Rsync can duplicate data between two directories whether the data is collocated on the same computer or over the network between two computers. The directories can be on any accessible random access storage devices (hard drives, thumb drives, etc). The data source must be readable and the destination must be writable.

The benefit of Rsync is the ability to update the mirrored backup with computed changes from the data source with a minimal load to system resources. It is very data bandwidth and system intensive to perform an entire copy of a directory structure each time a backup is to be performed. While the initial backup is no faster than a copy, Rsync has the ability to load the system in a minimal fashion by transferring only changes when subsequent rsync updates are performed.

Description of the rsync algorithm

Rsync client-server mirror between two computers:

Rsync command format: rsync [options] source-path destination-path

• Back-up (push) your photo directories to a second drive:
  rsync -av ~/Pictures /mnt/drive2
  This creates a backup of your photos in /mnt/drive2/Pictures/
  Back-up to a USB thumb drive is similar: rsync -av ~/Pictures /media/KINGSTON
  When you add new photos, just re-execute this rsync command to backup the latest changes.
  Note: The drive name will be dependent on the manufacturer.

  [Potential Pitfall]: Do not include the source directory name in the destination.
  rsync -av ~/Pictures /mnt/drive2/Pictures
  This will result in /mnt/drive2/Pictures/Pictures/
  Note that rsync destinations acts just like the cp and rcp commands.
  Also note that rsync -av ~/Pictures/ /mnt/drive2/ has a different behavior from rsync -av ~/Pictures /mnt/drive2

• Back-up (push) two directory source to a single directory on the second drive:
  rsync -av ~/Pictures ~/Images /mnt/drive2
  This creates a backup of your photos from the two directories in /mnt/drive2/Pictures/

• Sync directories and if any were deleted in ~/Pictures, also delete it in /mnt/drive2/Pictures/:
  rsync -a --progress --delete ~/Pictures /mnt/drive2
  This creates a backup in /mnt/drive2/Pictures/

• Sync one specific file in a directory:
  rsync -a ~/Pictures/Group-photo-2001-A.jpg /mnt/drive2/Pictures

• Sync a group of specific files in a directory:
  rsync -a ~/Pictures/2001-*.jpg /mnt/drive2/Pictures
  This creates a backup in /mnt/drive2/Pictures/
  Note that when transferring files only, the directory name has to be provided in the destination path.

• Sync files and directories listed in a file:
  rsync -ar --files-from=Filelist.txt ~/Data /mnt/drive2
  This creates a backup in /mnt/drive2/Data/
  Directory paths are included if specified with a closing slash such as pathx/pathy/pathz/. Path names must be terminated with a "/" or "/."

• Back-up (push) your source code and compress to save space. Ignore object files:
  rsync -avz --delete --exclude='*.o' --exclude='*.so' ~/src /mnt/drive2
  This creates a backup in /mnt/drive2/src/ but does not transfer files with the ".o" and ".so" extensions.

• Back-up (push) your source code and ignore object and shared object code files:
  rsync -av --delete --filter='+ *.[ch]' --filter='- *.o' ~/src /mnt/drive2
  This transfers files with the extension ".c" and ".h" but does not transfer object files with the ".o" extensions.
  same as rsync.exe -av --exclude='*.o' --filter='+ *.c *.h' ~/src /mnt/drive2

Command line argument	Description
-a (--archive)	Archive. Includes options: -r: recursion -l: preserve symbolic links as symbolic links. Opposite of -L -p: preserve permissions (Linux/unix only) -t: preserve file modification time -g: preserve group ownership -o: preserve user ownership -D: preserve special files and devices (Linux/unix only)
-d (--dirs)	Copy directory tree structure without copying the files within the directories
--existing	Update only existing files from source directory which are already present at the destination. No new files will be transferred.
-L (--copy-links)	Transform a symbolic link to a copied file upon transfer
--stats	Print verbose set of statistics on the transfer Add -h (--human-readable) to print stats in an understandable fashion
-p (--perms)	Preserve permissions (not relevant for MS/Windows client)
-r (--recursive)	Recursive through directories and sub-directories
-t (--times)	Preserve file modification times
-v (--verbose)	Verbose
-z (--compress)	Compress files during transfer to reduce network bandwidth. Files not stored in an altered or compressed state. Note that compression will have little or no effect on JPG, PNG and files already using compression. Use arguments --skip-compress=gz/bz2/jpg/jpeg/ogg/mp[34]/mov/avi/rpm/deb/ to avoid compressing files already compressed
--delete	Delete extraneous files from destination directories. Delete files on archive server if they were also deleted on client. Use the argument -m (--prune-empty-dirs) to delete empty directories (no longer useful after its contents are deleted)
--include --exclude --filter	Specify a pattern for specific inclusion or exclusion or use the more universal filter for inclusion (+)/exclusion (-). Do not transfer files ending with ".o": --exclude='*.o' Transfer all files ending with ".c" or ".h": --filter='+ *.[ch]'
-i (--itemize-changes)	Print information about the transfer. List everything (all file copies and file changes) rsync is going to perform
--list-only --dry-run	Don't copy anything, just list what rsync would copy if this option was not given. This helps when debugging the correct exclusion/inclusion filters.
--progress	Shows percent complete, Kb transferred and Kb/s transfer rate. Includes verbose output.

For all options see the rsync man page

1. connect client to a sever running rsync in daemon mode
2. connect client to a sever using a ssh shell

• Double colon refers to a connection to a host running the rsync daemon in the format hostname::module/path where the module name is identified by the configuration in /etc/rsyncd.conf. The double colon is equivalent to using the URL prefix rsync://
• Single colon refers to the use of a remote shell
• No colon then the directory is considered to be local to the system.

1) Rsync daemon server:

The Rsync server is often referred to as rsyncd or the rsync daemon. This is in fact the same rsync executable run with the command line argument "--daemon". This can be run stand-alone or using xinetd as is typically configured on most Linux distributions.

File: /etc/xinetd.d/rsync

Default: "disable = yes". Change to "disable = no"

service rsync { disable = no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID }

Start/Re-start xinetd: /etc/rc.d/init.d/xinetd restart

Typical Linux distributions do not pre-configure rsync for server use. Both Ubuntu and Red Hat based distributions require that one generates the configuration file "/etc/rsyncd.conf"

File: /etc/rsyncd.conf

log file = /var/log/rsyncd.log hosts allow = 192.17.39.244, 192.17.39.60 hosts deny = * list = true uid = root gid = root read only = false [Proj1] path = /tmp/Proj1 comment = Project 1 rsync directory [ProjX] path = /var/ProjX comment = Project X rsync directory

rsyncd.conf man page

Client command to rsync to the server:

Pull: rsync -avr server-host-name::Proj1 /home/user1/Proj1/Data
(eg. update mobile laptop from server backup)

2) Rsync to server using ssh shell:

First configure ssh for "password-less" login:
Note that current Linux distributions use ssh version 2 and rsa.

[user1@myclient ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/user1/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/user1/.ssh/id_rsa. Your public key has been saved in /home/user1/.ssh/id_rsa.pub. The key fingerprint is: aa:1c:76:33:8a:9c:10:51:............

Note that "Enter" was pressed when asked for a "passphrase" to take the default.
Two files are generated:

• Local client (private key): ~/.ssh/id_rsa
• Contents (one line) of file (public key): ~/.ssh/id_rsa.pub to be copied into file on server: ~/.ssh/authorized_keys

Note file protection on file:

[user1@myclient ~]$ ls -l ~/.ssh/id_rsa -rw-------. 1 user1 user1 1675 Sep 7 14:55 /home/user1/.ssh/id_rsa

Copy public key to server so you can login.
Use the following command:

[user1@myclient ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub user1@remote-server user1@remote-server's password:

Now try logging into the machine, with "ssh 'user1@remote-server'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting.

Test "password-less" ssh connection: ssh remote-server
This command should log you in without asking for a password.

rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/Data remote-server:/mnt/supersan/Proj1

Note that if this connection is is to be spawned by a cron job (eg. root user) then the shell user ID must be provided: user1@

rsync -avr --rsh=/usr/bin/ssh /home/user1/Proj1/Data user1@remote-server:/mnt/supersan/Proj1

SSH options may be put in the file ~/.ssh/config

crontab:

Note that rsync is often used with cron to perform a nightly rsync.

File: /etc/crontab

* 2 * * * rsync -avr server-host-name::Proj1/html /var/www > /var/log/rsync 2>&1

See the crontab man page

Multiple solutions exist for MS/Windows rsync clients but they all rely on Cygwin. There is no other port of rsync.

• Cygwin: Work in Cygwin shell environment
• DeltaCopy (client and server)
  Uses Rsync algorithm and protocol
• QtdSync: (client and server) MS/Windows version includes Cygwin rsync and ssh (and more)
  [download]

Cygwin must be installed for the purpose of this tutorial.
Be sure to install the Cygwin component "Net" + "rsync".

C:\cygwin\bin\rsync.exe \ssh.exe \sshpass.exe \cygcrypto-0.9.8.dll \cyggcc_s-1.dll \cygiconv-2.dll \cgypopt-0.dll \cygssp-0.dll \cygwin1.dll \cygz.dll

These components can be extracted for a minimal installation of be left in place as part of the complete Cygwin package.

@set PATH=C:\cygwin\bin @set LIB=C:\cygwin\bin rsync.exe -avr /cygdrive/c/MISC/JUNK remote-server::Proj1

[Potential Pitfall]: The following command: rsync.exe -avr c:MISC\JUNK remote-server::Proj1 will produce an error because rsync will interpret the "C:" as the remote host and not a drive letter. Drive letters are not POSIX compliant.

[Potential Pitfall]: The following commands:

cd c:\
rsync.exe -avr MISC\JUNK  remote-server::Proj1

For more on Cygwin POSIX paths: http://cygwin.com/cygwin-ug-net/using.html#using-pathnames

I chose QtdSync/cygwin as it provided a GUI and an isolated Cygwin rsync executable and its dependencies.

QtdSync GUI:

Includes rsync Cygwin components:

C:\Program Files\QtdSync\bin\rsync.exe \ssh.exe \sshpass.exe \cygcrypto-0.9.8.dll \cyggcc_s-1.dll \cygiconv-2.dll \cgypopt-0.dll \cygssp-0.dll \cygwin1.dll \cygz.dll \etc\fstab

Where the file C:\Program Files\QtdSync\etc\fstab has the following contents:

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

Note: QtdSync does not require the separate installation of cygwin.

@set PATH=C:\Program Files\QtdSync\bin @set LIB=C:\Program Files\QtdSync\bin rsync.exe -avr /cygdrive/c/MISC/JUNK remote-server::Proj1

Rsync daemon configuration requires installing rsync as a MS/Windows service. Use the Cygwin command cygrunsrv command (installed under the Cygwin "Admin" category).

Option	Description
-I --install	Install as a service
-R --remove	Remove as a service
-S --start	Start the service
-E --stop	Stop the service
-a --args	Arguments to pass the daemon service
-h	Command help instructions

Install rsync as a MS/Windows service: regrsyncd.bat

@set PATH=C:\cygwin\bin @set LIB=C:\cygwin\bin cygrunsrv -I rsyncd -e CYGWIN=nontsec --path /cygdrive/c/cygwin/bin/rsync.exe \ -a "--daemon --config=/etc/rsyncd.conf --no-detach"

You may be required to include the system user login and password:

cygrunsrv -I rsyncd -e CYGWIN=nontsec --path /cygdrive/c/cygwin/bin/rsync.exe \ -a "--daemon --config=/etc/rsyncd.conf --no-detach" -u Administrator -w super-secret-password

Note that Cygwin applications refer to configuration file referenced within the Cygwin environment. As viewed by the Cygwin rsync application, the file /etc/rsyncd.conf is actually C:\cygwin\etc\rsyncd.conf as viewed by the MS/Windows OS.

Where the option -e CYGWIN=nontsec turns off Microsoft Windows NT security permissions. On MS/Windows 7 use the following c:\cygwin\etc\fstab settings:

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

fstab man page

This is to avoid the following error:
rsync: failed to modify permissions on xxxxfilenamexxxx: Permission denied

File: C:\cygwin\etc\rsyncd.conf

log file = /var/log/rsyncd hosts allow = 192.17.39.244, 192.17.39.60 hosts deny = * list = true use chroot = false strict modes = false read only = false ignore nonreadable = yes dont compress = *.gz *.tgz *.zip *.rpm *.deb *.iso *.bz2 *.jpg *.mpg *.mpeg [Proj1] path = /cygdrive/c/Proj1 comment = Project 1 rsync directory

rsyncd.conf man page

Where the Cygwin log file /var/log/rsyncd is C:\cygwin\var\log\rsyncd as viewed by the MS/Windows OS.

The Cygwin rsync directory path /cygdrive/c/Proj1 is C:\Proj1\ as viewed by the MS/Windows OS.

Find "rsync". If not running, select the service and select "start" or issue the command "net start rsync".

[Potential Pitfall]: You must check to see if the service has started. If not select "Start the service".

Select "Control panel" + Windows Firewall":

To remove rsyncd from the available list of services, execute the following command from a "cmd" terminal:

c:\cygwin\bin\cygrunsrv -R rsyncd

Using telnet to test an rsync connection:

One can use telnet to test connectivity with an Rsync server:

[bash]$ telnet 192.168.1.12  873
Trying 192.168.1.12...
Connected to 192.168.1.12.
Escape character is '^]'.
@RSYNCD: 30.0
^C
Connection closed by foreign host.

This is the proper response when using telnet to connect to rsync.

This is representative of a telnet connection to a misconfigured rsync server or one which has not yet been started (nothing for the client to connect to):

[bash]$ telnet 192.168.1.13  873
Trying 192.168.1.13...
 telnet: connect to address 192.168.1.13: Connection timed out

does not get past this and times out.

Rsync client responses:

[bash]$ rsync -avr 192.168.1.12::Proj1 /home/user1/Test
rsync: failed to connect to 192.1.1.12: Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(124) [receiver=3.0.6]

Rsync logs:

2011/09/08 21:13:10 [1032] rsyncd version 2.6.6 starting, listening on port 873
2011/09/08 21:17:10 [3156] rsync error: received SIGUSR1 or SIGINT (code 20)

2011/09/08 21:19:10 [2796] rsync on Proj1/ from 192.168.1.10
2011/09/08 21:19:10 [2796] sent 940 bytes  received 334 bytes  total size 97

• Samba Rsync home page
• Lsyncd - Unix/Linux only. Monitor drive for changes (inotify) and perform an rsync any time a change is made.

• rsync.net
• s3rsync.com

Books:

"Backup & Recovery: Inexpensive Backup Solutions for Open Systems" by W. Curtis Preston ISBN # 0596102461, O'Reily Media (Jan 2007) Covers backup utilities including tar, dump, cpio, ntbackup, ditto, rsync, AMANDA, Bacula, BackupPC, rdiff-backup, and rsnapshot.