Showing total progress in rsync: is it possible?

Question

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the answers is 'no', but I'd like to see this confirmed.

Is it possible to show total file transfer progress with rsync?

Answer 1

danakim is correct. There are no trivial ways to add a total progress indicator.

The reason for this is that when rsync looks at a list of files to sync, it doesn't know ahead of time which files will need to change. If you are doing delta transfers, the deltas themselves have to be calculated ahead of time to give a total picture of the work that needs to be done.

In other words, the easiest way to calculate how much work there is to be done is to actually do it.

Answer 2

Basically no. You can only show progress per-file with the --progress flag, but that is about it.

I am guessing you can write a wrapper around it or use any of the patches that you already find but you must ask yourself if it is really worth it, do you actually need a total progress for rsync?

Answer 3

You can with 'pv' (apt-get install pv with Debian and ubuntu). I recommend to monitor the number of files transferred, since the amount of data transferred is not correlated to the size of files but on the delta between the source and destination. And counting files will count the same progress for one big delta and another one with small delta. Which means that in any case the ETA estimation might be far off. The size-based ETA only works if your destination is empty, in this case delta == size of source.

The general idea is to emit one line per file 'transferred' from rsync, and count those lines with 'pv':

rsync -ai /source remote:/dest | pv -les [number of files] >/dev/null

I tend to backup whole filesystems (for several reasons), in this case you can use the much cheaper df to get the number of files (rather than du or find wich will traverse your source hierarchy another time after rsync did it). The -x option appears to make sure rsync stays on the same source filesystem (and does not follow other inner mounts):

rsync -aix /source remote:/dest | pv -les $(df -i /source | perl -ane 'print $F[2] if $F[5] =~ m:^/:') >/dev/null

If you want to count files in /source in a general way, use find /source|wc -l (warning again: might be slow and heavy on I/O).

Answer 4

i used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

echo "Executing dry-run to see how many files must be transferred..."
TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

Answer 5

For long transfers, I'm happy with running du -s on both sides. Even wach -n1 du -s, if I feel really anxious.

Answer 6

Perhaps you can combine pv with rsync. Especially the parameter --size could by helpful. Taking a look at the docs, something like pv --size $(du -sb . | awk '{print $1}') | rsync -av . host:/your/path should work.

Here you'll find the docs and software.

Haven't tried this on my own.