In praise of rsync.

[nightflier]

A few years back, Bigpaws pointed me in the direction of rsync for performing data backup. Now I use this little gem all the time. Local file transfers, between machines on my LAN, through an SSH tunnel over the Internet, rsync does it all quickly, efficiently and reliably.

Latest success story I'd like to share: Got a computer in for repair. XP refusing to load. Tried a non-destructive reinstall, but the installer didn't even recognize old windows installation. A fresh install usually fixes stuff, but at the cost of having to re-install all your applications again. The box had MS Office and some other proprietary software, and of course the original install media/unlock codes were not with the machine. Besides, there were a bunch of customizations, bookmarks, the usual.

I booted a Linux Live CD and took a look. NTFS partition was there, files readable. I rsync'd everything from C: to my file server.

Next I ran the built-in, destructive recovery feature. Three hours later I had a new default OEM install of Windows. Another 45 minutes and it was cleaned up.

Then, I booted the Live CD again and reversed the copy process, overwriting the existing files on drive C:

I will admit my expectations were low, but windows loaded, with all the old settings intact. It ran slow, and I found that the filesystem had gotten very fragmented. I ran defrag, it was still running when I went to bed two hours later. Next morning it was done and the machine ran fine.

Open source empowers users.

[sledgehammer]

Nightflier,

Sometime ago, Bigpaws, I believe, suggested I consider rsync as an alternative to simplelinuxbackup.  It took a while, but now I use the Grsync root backup tool (on the 6.0 menu) instead of simplelinuxbackup.  I use Grsync to do a complete backup of the data partition on my laptop at least once a week. 

When (if) you get time, could you give a couple of examples of how you use rsync to transfer files between machines on a LAN?  I'm going to look into that as it could be very quite helpful. Right now, we copy to a usb stick or email.  Rather inconvenient and probably not all that secure.

John

[hata_ph]

Ya, I also am interested to learn about rsync...

[nightflier]

For quick local operations, the basic syntax is:

Code:

rsync -av /path/to/source/ /path/to/destination

The "a" option specifies "archive" mode, which preserves permissions, "v" gives more feedback.
Adding a trailing slash to "source" avoids creation of a "source" sub-dir in "destination".
In the above described rescue operation I first mounted my file server using cifs.

rsync is best for directories, not individual files. To copy a single file, use scp:

Code:

scp /path/to/sourcefile username@remote-macine:/path/to/destination

or the other way:

Code:

scp username@remote-macine:/path/to/sourcefile /path/to/destination

My desktop is a little non-standard: http://forum.vectorlinux.com/index.php?topic=4564.0 I use my central file server a lot. Big files go over the network mountpoints to the server, and scripts sync my home dir with a copy that resides on a server drive.

This example illustrates how I use rsync to back up my home dir:

Code:

#!/bin/sh
# Backs up home dir to server. Runs every hour from crontab
rsync -auz -e "ssh -i /home/username/.ssh/username-rsync-key" --exclude-from /home/username/scripts/exclude --delete /home/username/  192.168.1.X:/home/username

- This archives all files with permissions intact (a), updates newer files (u), uses compression (z),
- connects using ssh and a key without a password * (see below),
- excludes mount points that map to the server, using a simple text file:

Code:

Music
Storage

- deletes extraneous files on server.

Another script shuts the machine down:

Code:

#!/bin/sh
# Opens a terminal, runs backup, then shuts down
konsole -e rsync -auz -e "ssh -i /home/username/.ssh/username-rsync-key" --exclude-from /home/username/scripts/exclude --delete /home/username/  192.168.1.X:/home/username

sudo /sbin/shutdown -h now

Adding the "konsole -e" gives me a terminal window so I can see the progress.

* For automated network operations, using ssh without first mounting the network resources, you first need to configure your machines for access using password-less public/private key pairs. I relied heavily upon this guide:
http://troy.jdmz.net/rsync/index.html
For backup through the internet I implemented the restrictions script, but not for my local network.

[sledgehammer]

Thanks!

I have bookmarked this page and will return to it often.

John

[Windozer]

Hi Nightflier,

do you think this would work for backing up temporarily mounted NTFS partitions?  I've got a mish-mash of XP and Linux boxes. At this point, the XP stuff is automated in the usual way w/NTbackups to a server box. But for the Linux stuff I just let cron kick off a simple copy of what I need ... it would be nice to be able to have one thing do both.  Here's what concerns me:

1. having to jump through more SAMBA hoops
2. Would the NTFS stuff restore properly after the image was saved on say Reiserfs?

I've put ghosted XP drive images on and off of Reiser before, but it seemed weird somehow... it worked...

Am I choosing a more difficult route trying to us rsync? 

thanks
- Howard
PS: I think your non-standard tempfs system is really cool!

[bigpaws]

It would work with any mounted filesystems. You would just need to inform
rsync when the drives were mounted.

I use rsync in cygwin as well as on Linux. My clients have off site and on site backups
being preformed with rsync. The only time I have had a failure is when there is no
connection.

Thanks for the praise guys.

Bigpaws