Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've only just started trying to use rsync to do some backups from a couple of Linux machines to a Windows server, which runs the rest of the backup system. After a whole lot of reading and experimenting I finally got it all working on one of the Linux machines (Centos), so I copied the script over to the other machine (Debian), edited some paths and tried to run it. Instead of running the way it did on the first machine I get the following result:

building file list ...
1 file to consider

sent 36 bytes  received 16 bytes  11.56 bytes/sec
total size is 0  speedup is 0.00

The script in question is:

#! /bin/sh
/usr/bin/rsync \
--progress \
--recursive \
-t \
--delete-excluded \
--filter="- .cpan/***" \
--filter="- include/***" \
--filter="- cache/***" \
--filter="- lib/***" \
--filter="- cvs/***" \
--filter="+ /BayesTraining/***" \
--filter="+ /BogoTraining/***" \
--filter="+ /etc/***" \
--filter="+ /var/***" \
--filter="+ /root/***" \
--filter="+ /boot/***" \
--filter="+ /usr/local/***" \
--filter="+ /home/***" \
--filter="- *" \
/ \
rsync://myserver/targetfolder

It will only start sending files if I remove that last filter parameter but then it sends a whole bunch more than I want.

Where have I stuffed up and what do I need to change? (Before anyone asks, I'm not using rsync over SSH because the Windows server does not yet have an SSH service on it yet. Small steps...)

Edit: rsync --version reveals

Centos: rsync  version 2.6.8  protocol version 29
Debian: rsync  version 2.6.4  protocol version 29

Solution: Thanks to pavium for pointing out that I have overlooked a crucial part in the man page (reformatted for readability):

a trailing "dir_name/***" will match both the directory (as  if "dir_name/"
had been specified) and everything in the directory (as if "dir_name/**"
had been  specified). This  behavior  was added in version 2.6.7.

The solution is to create the following rules for each directory I want included:

--filter="+ /dirname/"
--filter="+ /direname/**"
share|improve this question
    
On a side note, you should upgrade your rsync versions. There have been a lot of bug fixes and performance improvements. –  David Jan 6 '10 at 4:45
    
I agree. I'm just waiting for the next stable release of Mailcleaner, which is what the Debian machine is for. That will bring it up to date. –  John Gardeniers Jan 6 '10 at 8:21

3 Answers 3

up vote 3 down vote accepted

The last filter item may be your problem (well, I think you know that). I found the following in the rsync 2.6.9 man page:

Note that, when using the --recursive (-r) option (which is implied by -a), every subcomponent of every path is visited from the top down, so include/exclude patterns get applied recursively to each subcomponent’s full name (e.g. to include “/foo/bar/baz” the subcomponents “/foo” and “/foo/bar” must not be excluded). The exclude patterns actually short-circuit the directory traversal stage when rsync finds the files to send. If a pattern excludes a particular parent directory, it can render a deeper include pattern ineffectual because rsync did not descend through that excluded section of the hierarchy. This is particularly important when using a trailing ’*’ rule. For instance, this won’t work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory “some” is excluded by the ’*’ rule, so rsync never visits any of the files in the “some” or “some/path” directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: “+ */” (put it somewhere before the “- *” rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

So, the - * is excluding paths not explicitly included earlier in the filter rules. The man page also tells me that the trailing "dir_name/***" behaviour (which would have enabled those paths) was added at version 2.6.7 (your Debian is earlier)

share|improve this answer
    
"was added at version 2.6.7 (your Debian is earlier)". Don't ask me how I overlooked that. Seems that's where I need to make changes. Thanks. –  John Gardeniers Jan 6 '10 at 1:03

Psychic debugging mode activated:

Bash is probably expanding the * into a list of all files in the current directory, so rsync doesn't see a * it sees file1 file2 file3.

Try escaping the * - --filter="- \*" \, or --filter='- *' \ should do the trick.

share|improve this answer
    
If * doesn't actually expand to anything - say, because you're in a folder that has no contents - bash will pass the * along unaltered –  James Polley Jan 5 '10 at 23:38
    
mwahahaha. Psychic debugging SUCCESS! At least, I assume the big green tick means that I was on the right track :) –  James Polley Jan 6 '10 at 0:12
    
Bingo! Running as I type. Thank you. Now I'd like to understand why an asterisk needs to be escaped when it's inside quotes. Plus, why it only needs to be escaped on the Debian box and not on Centos. –  John Gardeniers Jan 6 '10 at 0:12
    
I responded too early. It's running alright but ignoring that last filter ruls, so is again copying more than I asked for. :( –  John Gardeniers Jan 6 '10 at 0:14
    
but happily, Pavium seems to have the answer :) –  James Polley Jan 6 '10 at 1:25

Run 'rsync --version' on your CentOS and Debian boxes, and compare the results. Is the Debian one older perhaps? It's possible it's a fault of that particular version or the way it was compiled on that box.

share|improve this answer
    
Version information added. In both cases rsync is whatever came on the installation CD, although the Debian machine came pre-configured as a mail scanner (Mailcleaner). –  John Gardeniers Jan 5 '10 at 23:21
1  
rsync.samba.org/ftp/rsync/ChangeLog.gz Browsing the Changelog post 2.6.4 (around Mar 31 2005) I see several mentions of changes to the way wildcards are globbed when passed as options; as James mentions it could be the way in which either the bash or rsync on Debian is globbing your '*' that has been fixed/updated/changed on the CentOS box which appears to be V5, the newest. –  troyengel Jan 5 '10 at 23:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.