If you have been working with Linux for any amount of time, you must be familiar with the rsync command utility. rsync is a very powerful tool that allows you to synchronize files between two locations, even remote locations over the network. It is a very powerful file transfer tool with definite advantages over any other copy programs.
The generic syntax of rsync command is
rsync [options...] src [dest]
The src and dest stands for source and destination folders (or files) respectively. You can access remote shells for either the src or dest using the following format.
rsync [options...] [user@]host:src dest
rsync [options...] src [user@]host:dest
We will see several examples that will demonstrate how the rsync can be used for different scenarios. Before we start with the examples, there are some useful options that you should know that we will throughout the examples.
-a or –archive: this is short for a set of other options. known as the archive mode, it is short for -rlptgoD.
-r or –recursive: if it is directory, recurse into sub-directories
-l or –links: preserve links. it copies symlinks as symlinks.
-p or –perms: preserve permissions of the files and folders
-t or –times: this preserves the timestamps of the files and folders
-g or –group: preserves the user groups
-o or –owner: preserves the owner
-D : preserves devices and special files
-z or –compress: compress file data when transferring files
–delete: delete extraneous files at the destination
-v or –verbose: verbose mode
transferring a single file
If you want to copy or synchronize a single file between two local locations:
$ rsync -avz /home/tom/directory1/file1.txt /home/tom/directory2/
Although, the recursive options are not needed in this example, we will use the easy to use -a option and it will not have any effect on this example, other than to preserve the file meta data such as groups, owner, time etc.
$ rsync -avz tom@remotemachine:/home/tom/dir1/file1.txt /home/james/dir2/
The above example copies the file (file1.txt) from a remote machine to the local folder. For the reminder of the post, we will assume that you are copying between two local folders (unless the example warrants a remote location). You can easily replace either the source or the destination in the examples with remote machine information if needed.
transferring multiple files
$ rsync -avz ~/file1.txt ~/file2.txt ~/file3.txt ~/dir1/ /home/james/dir2/
You can copy multiple files and folders from different locations into the same destination.
transferring a single directory (and sub-directories)
Now, let’s try to copy a folder and all its contents including the sub-folders. In this example, the source ends with a slash (/) which is the most common usage.
$ rsync -avz /home/tom/dir1/ /home/james/dir2/
This copies all the files from the folder dir1/ to the folder dir2/
$ rsync -avz /home/tom/dir1 /home/james/dir2/
Notice the missing slash at the end of the source in the example. The above command will copy the all files including the directory dir1 into the dir2/ folder. There will a new folder named dir1 inside dir2 after this executes.
Sometimes, you don’t want to copy all the files but only specific files that match a pattern. Let’s say we find to copy all log files (with extension .log)
$ rsync -avz /home/tom/dir1/*.log /home/tom/dir2/
include files by pattern
You can include files explicitly using the –include option or the shortcut +. The first matching rule takes effect, which could be either an include or exclude pattern. rsync uses a slightly modified and simpler pattern matching format than regular expression, but is pretty similar.
$ rsync -avz --include=**/*.log --exclude=* ~/dir1/subdir1/ ~/dir2/
exclude files by pattern
$ rsync -avz --exclude=**/*.log ~/dir1/subdir1/ ~/dir2/
The above example copies all files, but excluding the files with a file extension .log.
exclude/include files by size
$ rsync -avz --max-size=100M ~/dir1/subdir1/ ~/dir2/
The command line option –max-size will allow you to specify the maximum size of the file that you want transferred. The above example will not transfer any files that are more than 100 Mb in size. There is also an option to specify the minimum size just the same way….
$ rsync -avz --min-size=100M ~/dir1/ ~/dir2/
This will ensure that only files that are greater than 100 Mb in size are transferred.
copy to remote server
We touched on this in some examples a little earlier in the post. Here is an example to copy files to a remote server.
$ rsync -avz ~/dir1/ user@remotemachine:/home/tom/dir1/
copy from a remote server
Similar to the earlier example, copying from a remote server is just as easy.
$ rsync -avz user@remoteserver:/home/tom/dir1/ ~/dir1/
deleting files at destination
The rsync is primarily designed to keep the files synchronized, which means it has to also keep the deleted files synchronized. If a file has been deleted at the source since last time the command has been run, the command can delete it from the target as well.
$ rsync -avz --delete ~/dir1/subdir1/ ~/dir2/
There are other related options such as –delete-after, –delete-during and –delete-before, which determines when the files are deleted in the transfer process.
You can choose to do a transfer that copies files that are updated on source and skip the ones that are newer on the destination or receiver. The –update or -u command line option will allow you to do exactly this. This allows you to preserve changes that you have made on the target or destination.
$ rsync -avz -u --delete ~/dir1/subdir1/ ~/dir2/
dry run option
When running a rsync command with several different options and regular expressions, you might want to make sure that the command is “correct” and transfers the files just as you intended. You can use the –dry-run (or -n) option which basically will display what it will do, but will not actually perform any action.
$ rsync --dry-run -avz --delete ~/dir1/subdir1/ ~/dir2/
Once you have verified that the files, regular expressions and options are all correct, you can remove the –dry-run option. If you always use the same preset options (such as -avz) as I do, then it is a perfect candidate for bash alias. Create bash aliases for some of the commonly used commands which will make it even easier to execute.