Showing total progress in rsync: is it possible?

Question

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the answers is 'no', but I'd like to see this confirmed.

Is it possible to show total file transfer progress with rsync?

Answer 1

danakim is correct. There are no trivial ways to add a total progress indicator.

The reason for this is that when rsync looks at a list of files to sync, it doesn't know ahead of time which files will need to change. If you are doing delta transfers, the deltas themselves have to be calculated ahead of time to give a total picture of the work that needs to be done.

In other words, the easiest way to calculate how much work there is to be done is to actually do it.

Answer 2

Basically no. You can only show progress per-file with the --progress flag, but that is about it.

I am guessing you can write a wrapper around it or use any of the patches that you already find but you must ask yourself if it is really worth it, do you actually need a total progress for rsync?

Answer 3

For long transfers, I'm happy with running du -s on both sides. Even watch -n1 du -s, if I feel really anxious.

watch executes a command (du -s here) periodically (every 1 second here) and shows the output fullscreen.

Answer 4

Perhaps you can combine pv with rsync. Especially the parameter --size could by helpful. Taking a look at the docs, something like pv --size $(du -sb . | awk '{print $1}') | rsync -av . host:/your/path should work.

Here you'll find the docs and software.

Haven't tried this on my own.

Answer 5

You can with 'pv' (apt-get install pv with Debian and ubuntu). I recommend to monitor the number of files transferred, since the amount of data transferred is not correlated to the size of files but on the delta between the source and destination. And counting files will count the same progress for one big delta and another one with small delta. Which means that in any case the ETA estimation might be far off. The size-based ETA only works if your destination is empty, in this case delta == size of source.

The general idea is to emit one line per file 'transferred' from rsync, and count those lines with 'pv':

rsync -ai /source remote:/dest | pv -les [number of files] >/dev/null

I tend to backup whole filesystems (for several reasons), in this case you can use the much cheaper df to get the number of files (rather than du or find wich will traverse your source hierarchy another time after rsync did it). The -x option appears to make sure rsync stays on the same source filesystem (and does not follow other inner mounts):

rsync -aix /source remote:/dest | pv -les $(df -i /source | perl -ane 'print $F[2] if $F[5] =~ m:^/:') >/dev/null

If you want to count files in /source in a general way, use find /source|wc -l (warning again: might be slow and heavy on I/O).

Answer 6

i used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

echo "Executing dry-run to see how many files must be transferred..."
TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

Answer 7

It seems that soon there will be an official solution: following the advice in this question I tried the development version of rsync.

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for tranferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

Answer 8

I used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

#echo "Executing dry-run to see how many files must be transferred..."
#TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')
TODO=$(find ${SOURCES} | wc -l)

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

I changed the TODO dry-run to

TODO=$(find ${SOURCES} | wc -l)

It finds the number of files very fast!

Answer 9

I'd make this a comment but don't have enough reputation. In response to naught101's comment to the chosen answer, the --progress option shows how many files have been transfered out of the total amount to transfer. I didn't realize this until looking at this post and looking at the output more carefully.

The 'to-check' stat shows how many files are left out of the total. This is of most use when rsync'ing to a new destination so you know all files will be fully copied.

From the man page:

      When [each] file transfer  finishes,  rsync  replaces  the
      progress line with a summary line that looks like this:

           1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

      In this example, the file was  1238099  bytes  long  in
      total,  the average rate of transfer for the whole file
      was 146.38 kilobytes per second over the 8 seconds that
      it took to complete, it was the 5th transfer of a regu-
      lar file during the current rsync  session,  and  there
      are 169 more files for the receiver to check (to see if
      they are up-to-date or not) remaining out  of  the  396
      total files in the file-list.