The Rsync command is a popular utility for synchronizing files and directories between two remote systems.
Unlike other utilities, Rsync is highly recommended for transferring large amounts of data over a network. That’s because it can efficiently transfer only the portions of files that have changed rather than transferring the entire file each time.
Some of the popular Rsync features include
- Efficiency: As mentioned, Rsync can transfer only the parts of a file that have changed, which can significantly reduce the amount of data that needs to be transferred.
- Flexibility: Rsync has a wide range of options that allow users to customize its behavior to fit their specific needs. For example, users can specify the type of files to be transferred, exclude specific files or directories from being transferred, and control how Rsync handles file permissions and ownership.
- Versatility: Rsync can be used to transfer files over a network using various protocols, including ssh, rsh, and rsync itself. It can also be used to synchronize files between local directories.
- Wide availability: Rsync is available for different platforms, including Linux, macOS, Single Board devices (Raspberry Pi, etc.), and Windows.
Rsync Command Skipping Directories – Error
If you have been working with the Rsync command for a while, you have likely encountered the error “skipping directory.” Rsync throws this error on the Terminal when it can’t sync/copy a specified directory. Take a look at the example below.
We have a “backup” directory that we wish to copy to our remote server – “server-one.” Unfortunately, when we executed the command below on our Terminal, we got the error “skipping directory.”
rsync -v backup/ [email protected]:/home/server-one/
See the image below.
When we ran the ls command on the server, we saw that neither the directory nor the files inside the “backup” directory were synced/copied. So, why did Rsync skip this particular directory? What is the solution to such an error?
Solutions
There are several possible reasons why the Rsync command might skip directories. This post will look at each scenario and recommend a solution.
Not syncing/ copying a Directory Recursively (-r or -a).
“Recursive” is a popular term you will always come across when working with files and directories on the Linux Terminal. But what exactly does it mean?
When you add the recursive option (-r) in the Rsync syntax, that means the command will copy the directory itself and all of the files and subdirectories inside it, all of the files and subdirectories inside those subdirectories, and so on. In other words, the Rsync command will traverse the directory tree recursively and copy everything it finds.
However, the Rsync command will skip that directory if you don’t include the -r parameter when copying a directory. Take a look at the example below.
rsync -v backup/ [email protected]:/home/server-one/
After running the command above, we got the error shown in the image below.
This error occurred because we tried to copy a directory without the recursive (-r) parameter. The solution is to add the -r parameter in the Rsync command. See below.
rsync -v -r backup/ [email protected]:/home/server-one/
When you look at the image below, you can see that we did not get any error, and the files inside the directory were copied successfully.
Tip: Note that the Rsync command also has a -a option, which is equivalent to -dpR, and which combines the -d, -p, and -R options. The -d option tells the Rsync command to preserve symbolic links, the -p option tells the cp command to preserve the file attributes of the original files (such as ownership and permissions), and the -R option is equivalent to the -r option and tells the Rsync command to copy directories and their contents recursively.
Using Filtering Options (--include, --exclude, and --filter).
The --include, --exclude, and --filter options allow you to specify patterns for including or excluding files and directories from the transfer. If a directory doesn’t match any of the specified patterns, it will be skipped. Let’s have a more in-depth look using the --exclude option.
The --exclude option in the Rsync command allows you to specify patterns or files that should be excluded from the synchronization process. This can be useful if you want to exclude certain types of files or directories from being transferred or if you want to exclude specific files or directories by name.
To use the --exclude option, you will need to specify the pattern or name of the file or directory that you want to exclude. For example:
rsync -av --exclude 'test-directory' /source/ /destination/
This command will synchronize the contents of the /source/ directory with the /destination/ directory, but it will the test-directory from the synchronization process. Take a look at the scenario below.
We want to copy the files and directories inside the “Test-Directory” to our remote server. However, we want to exclude the “dir-1” directory. We will use the command below.
rsync -av --exclude 'dir-1' Test-Directory/ [email protected]:/home/server-one/
After running the command above, Rsync will copy all the files and folders except the “dir-1” directory. See the image below.
Permission issues
Rsync will skip a directory if it doesn’t have the necessary permissions to access it. By default, Rsync will not copy files or directories for which it does not have read permissions.
If you want Rsync to copy these files or directories, you can use the --ignore-perms option. This will cause Rsync to ignore permissions and copy the files or directories regardless of whether it has the necessary permissions.
However, it is generally not a good idea to use this option unless you are sure you want to ignore permissions and understand the implications of doing so.
Using an Outdated File List with the --delete Option
The --delete option tells Rsync to delete any files or directories at the destination that are not present at the source. This can be useful for ensuring that the destination is an exact copy of the source, but it can also lead to unexpected results if the file list used by Rsync is outdated.
The --delete option can cause Rsync to skip a directory if it is not present in the source directory and the file list used by Rsync is outdated.
For example, if you run Rsync with the --delete option, and the source directory contains a file or directory that has since been deleted, Rsync will delete the corresponding file or directory at the destination as well.
Similarly, if you run Rsync with the --delete option, and the file list used by Rsync is outdated such that it includes a directory that no longer exists in the source, Rsync will skip that directory and not delete it at the destination.
In general, it is a good idea to be cautious when using the --delete option, as it can lead to data loss if used improperly. It is always a good idea to make sure that you have a backup of your data before using this option, just in case.
Symbolic Links
A symbolic link, also known as a symlink or soft link, is a special type of file that points to another file or directory on a computer system. When you access a symbolic link, you effectively access the file or directory it points to.
The --copy-links option tells Rsync to follow symbolic links and transfer the files they point to rather than transferring the symbolic links themselves. This means that if you have a symbolic link pointing to a file, Rsync will transfer the actual file rather than the symbolic link.
The --copy-unsafe-links option is similar to --copy-links, but it also allows Rsync to follow symbolic links that point to files or directories outside the source and destination directories. This can be useful if you want to transfer files that are linked to by symbolic links in different locations on the file system.
However, it also raises one major problem. If a symbolic link points to a directory that doesn’t exist, Rsync will skip it. This means that Rsync will not follow the link or transfer any files that are linked to by the symbolic link.
Conclusion
This post has given you a comprehensive guide on why the Rsync command is skipping certain directories.
If you are experiencing this issue and are not sure what is causing it, you can try adding the --verbose or -v option to your Rsync command to see a detailed log of what Rsync is doing. That will greatly help to identify the cause of the problem.
Do you have any comments or queries regarding this post? Please don’t hesitate to let us know in the comments.
