Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the answers is 'no', but I'd like to see this confirmed.

Is it possible to show total file transfer progress with rsync?

share|improve this question
1  
Since rsync 3.1.0 there is now a basic total file transfer progress included. See answer serverfault.com/a/441724/107793 for details. –  Florian Feldhaus Jan 14 at 15:14

13 Answers 13

up vote 18 down vote accepted

danakim is correct. There are no trivial ways to add a total progress indicator.

The reason for this is that when rsync looks at a list of files to sync, it doesn't know ahead of time which files will need to change. If you are doing delta transfers, the deltas themselves have to be calculated ahead of time to give a total picture of the work that needs to be done.

In other words, the easiest way to calculate how much work there is to be done is to actually do it.

share|improve this answer
5  
You could still have a simple indicator like (data transferred + data skipped)/(total data in source), or (# files transferred or skipped)/(# files in source). It's not going to be particularly accurate, but it'd give an idea. Good for when you're doing a big transfer at the end of the day, and you're wondering whether to wait around and turn off the computer, or let it run for the night... –  naught101 May 29 '12 at 6:52
    
I don't believe that conclusion is right. I think @naught101 is being more fair to the question, and I think predicating an answer on not using --size-only or the like is further inaccurate. –  Evan Carroll May 24 at 0:11
    
At the time when I wrote the reply, it was accurate -- rsync did not have a mechanism for a total progress indicator. And yes, you could write your own, but very few people would. –  David Mackintosh Jun 16 at 21:11

It seems that soon there will be an official solution: following the advice in this question I tried the development version of rsync.

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for tranferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

share|improve this answer
1  
Note that this has landed in 3.1.0. It is also important to note that this isn't necessarily accurate in the time perspective. It is essentially showing the amount of data that has been verified to exist on the remote end. And the rate is the rate at which the data is being learned to be correct on the remote end (whether it was already that way or the new data was transferred and made it correct). So although very useful you need to understand the caveats. –  Kevin Cox Nov 28 '13 at 21:11

You can with 'pv' (apt-get install pv with Debian and ubuntu). I recommend to monitor the number of files transferred, since the amount of data transferred is not correlated to the size of files but on the delta between the source and destination. And counting files will count the same progress for one big delta and another one with small delta. Which means that in any case the ETA estimation might be far off. The size-based ETA only works if your destination is empty, in this case delta == size of source.

The general idea is to emit one line per file 'transferred' from rsync, and count those lines with 'pv':

rsync -ai /source remote:/dest | pv -les [number of files] >/dev/null

I tend to backup whole filesystems (for several reasons), in this case you can use the much cheaper df to get the number of files (rather than du or find wich will traverse your source hierarchy another time after rsync did it). The -x option appears to make sure rsync stays on the same source filesystem (and does not follow other inner mounts):

rsync -aix /source remote:/dest | pv -les $(df -i /source | perl -ane 'print $F[2] if $F[5] =~ m:^/:') >/dev/null

If you want to count files in /source in a general way, use find /source|wc -l (warning again: might be slow and heavy on I/O).

share|improve this answer

Basically no. You can only show progress per-file with the --progress flag, but that is about it.

I am guessing you can write a wrapper around it or use any of the patches that you already find but you must ask yourself if it is really worth it, do you actually need a total progress for rsync?

share|improve this answer

i used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

echo "Executing dry-run to see how many files must be transferred..."
TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"
share|improve this answer

For long transfers, I'm happy with running du -s on both sides. Even watch -n1 du -s, if I feel really anxious.

watch executes a command (du -s here) periodically (every 1 second here) and shows the output fullscreen.

share|improve this answer
1  
Thanks for the example of the watch command! –  Cam Sep 17 '12 at 4:40

I used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

#echo "Executing dry-run to see how many files must be transferred..."
#TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')
TODO=$(find ${SOURCES} | wc -l)

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

I changed the TODO dry-run to

TODO=$(find ${SOURCES} | wc -l)

It finds the number of files very fast!

share|improve this answer
    
find works so much better than rsync --dry-run! –  hopeseekr Jan 5 '13 at 2:11
2  
find only works if you're rsync-ing locally. rsync --dry-run works with remote sources too... –  voretaq7 May 8 '13 at 17:29

I'd make this a comment but don't have enough reputation. In response to naught101's comment to the chosen answer, the --progress option shows how many files have been transfered out of the total amount to transfer. I didn't realize this until looking at this post and looking at the output more carefully.

The 'to-check' stat shows how many files are left out of the total. This is of most use when rsync'ing to a new destination so you know all files will be fully copied.

From the man page:

      When [each] file transfer  finishes,  rsync  replaces  the
      progress line with a summary line that looks like this:

           1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

      In this example, the file was  1238099  bytes  long  in
      total,  the average rate of transfer for the whole file
      was 146.38 kilobytes per second over the 8 seconds that
      it took to complete, it was the 5th transfer of a regu-
      lar file during the current rsync  session,  and  there
      are 169 more files for the receiver to check (to see if
      they are up-to-date or not) remaining out  of  the  396
      total files in the file-list.
share|improve this answer

I also searched for how to show the total progress with rsync and I've found a useful answser from this post: http://stackoverflow.com/questions/7157973/monitoring-rsync-progress

Basicly, you can use --info=progress2 in the dev version of rsync 3.1.0. Here is what the doc has said:

There is also a --info=progress2 option that outputs statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify --info=name0 if you want to see how the transfer is doing without scrolling the screen with a lot of names. (You don't need to specify the --progress option in order to use --info=progress2.)

share|improve this answer

Perhaps you can combine pv with rsync. Especially the parameter --size could by helpful. Taking a look at the docs, something like pv --size $(du -sb . | awk '{print $1}') | rsync -av . host:/your/path should work.

Here you'll find the docs and software.

Haven't tried this on my own.

share|improve this answer

Use

lsof -ad3-999 -c rsync

To see what files rsync currently has open (will show filesize) rsync copies into a hidden file locally

share|improve this answer

Might be a bit late here but future answer-seekers might benefit.

This was bugging me too so I thought I'd get down and dirty and write my first script. The package zenity must be installed (sudo apt-get install zenity), but I'm sure it will probably be there already. Also, I use wmctrl (window manager control) to change the title of the progress dialog when it's done, it's easily installable but won't make a difference if you don't. I just like to see when it's done in my panel.

The script basically asks for a source and destination directory, calculates the percentage of the destination over the source in size using du and displays a progress bar.

Note: This only works for complete directory/file syncing (I usually use it to make backups of apt cache), so no --exclude=/file/in/Source-directory option. Also won't work if there are files/directories in Destination directory not in the source directory. I'm not sure if it works for remote sources/destinations since never I've had the need for it or the resources to test it.

PS. This script might be very badly written or very inefficient, (script-virgin here), but at least it serves it's purpose and of course you're welcome to edit and improve it to suit your needs. PSS. Also, couldn't get the cancel button to kill rsync so I just removed it.

    #!/bin/bash
set -e;

WELC="Running RsyncP as $USER";

function echo_progress()
{
    while (($TRANSFER_SIZE > 1000));    
    do  
        DEST_SIZE=$(du -s $DEST_FOLDER | cut -d / -f 1); 
        ((TRANSFER_SIZE=$SOURCE_SIZE-DEST_SIZE)); 
        PROGRESS_PERC=$((DEST_SIZE*100/SOURCE_SIZE));
        echo $PROGRESS_PERC;
        sleep 0.1s;
    done;
    echo 100;
    zenity --info --width=250 --title=RsyncP --text="File syncing complete!";
}

function get_input()
{
    dirs=$(zenity --forms --width=500 --title="RsyncP" --text="Enter source And destination directories" --add-entry="Source: " --add-entry="Destination: " --separator=" ");

    SOURCE_FOLDER=$(echo $dirs | cut -d' ' -f 1);
    DEST_FOLDER=$(echo $dirs | cut -d' ' -f 2);

    OPTIONS=-$(zenity --list --title="RsyncP Options" --text="Select rsync options" --separator='' --height=470 --width=470 --checklist --column "activate" --column "Option" --column "Description" FALSE v "Verbose (Terminal only)" FALSE q "Quiet, supress non-error messages (Terminal only)" FALSE P "Progress (Terminal only)" FALSE a "Archive (lrpog)" TRUE r "Recurse into directories" FALSE p "Preserve permissions" FALSE o "Preserve owner" FALSE g "Preserve group" FALSE l "Copy symlinks as symlinks");

    zenity --question --no-wrap --title="RsyncP" --width=500 --text="rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER\nDo you want to continue?";

    SOURCE_SIZE=$(du -s $SOURCE_FOLDER | cut -d / -f 1); 
    DEST_SIZE=$(du -s $DEST_FOLDER | cut -d / -f 1); 
    PROGRESS_PERC=$((DEST_SIZE*100/SOURCE_SIZE)); 
    TRANSFER_SIZE=1001;
}

if [ "$(id -u)" != "0" ]; then
    zenity --question --title=RsyncP --text="$WELC, Continue?";
    get_input;
    rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER & 
    echo_progress | zenity --progress --title=RsyncP --no-cancel --auto-close --text="Copying from \n$SOURCE_FOLDER to \n$DEST_FOLDER" ;

else            
    zenity --question --title=RsyncP --text="$WELC, Continue?";
    get_input; 
    sudo rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER & 
    echo_progress | zenity --progress --title=RsyncP --no-cancel --auto-close --text="Copying from \n$SOURCE_FOLDER to \n$DEST_FOLDER" ;
fi
share|improve this answer

Note the caveat here that even --info=progress2 is not entirely reliable since this is percentage is based on the number of files rsync 'knows' about at the time when the progress is being displayed. This is not necessarily the total number of files that needed to be sync'd (for instance, if it discovers a large number of large files in a deeply nested directory). One way to ensure that --info=progress2 doesn't 'jump back' in the progress indication would be to force rsync to scan all the directories recursively before starting the sync (instead of its default behavior of doing an incrementally recursive scan), by also providing the --no-inc-recursive option. Note however that this option will also increase rsync memory usage and run-time.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.