383

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the answers is 'no', but I'd like to see this confirmed.

Is it possible to show total file transfer progress with rsync?

3

20 Answers 20

596

There is now an official way to do this in rsync (version 3.1.0 protocol version 31, tested with Ubuntu Trusty 14.04).

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for transferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

References:

15
  • 23
    Note that this has landed in 3.1.0. It is also important to note that this isn't necessarily accurate in the time perspective. It is essentially showing the amount of data that has been verified to exist on the remote end. And the rate is the rate at which the data is being learned to be correct on the remote end (whether it was already that way or the new data was transferred and made it correct). So although very useful you need to understand the caveats.
    – Kevin Cox
    Nov 28, 2013 at 21:11
  • 31
    Worth noting that --info=progress2 doesn't work with -v
    – sanmai
    Mar 8, 2015 at 15:09
  • 10
    @sanmai: Yes it does! It displays verbose output (which file is being worked on at the current time), but instead of showing the relative progress percentage for that individual file, it shows the absolute progress of the entire rsync operation. May 2, 2015 at 19:16
  • 77
    Add the --no-i-r switch aswell, so rsync does not scan incrementally but completely before copying and knows+displays how much work is left.
    – Alex
    Nov 19, 2015 at 13:17
  • 34
    Note that you can use --human-readable (or -h) to see total size in MB/GB.. in progress.
    – Nux
    Aug 23, 2017 at 10:22
107

The following applies to rsync version 3.0.0 and above. The options described below were introduced in that release on March 1st, 2008.

Along with --info=progress2 you can also use --no-inc-recursive option (or its shorter --no-i-r alias) to disable incremental recursion.

This will build the entire file list at the beginning, rather than incrementally discovering more files as the transfer goes on. Since it will know all files before starting, it will give a better report of the overall progress. This applies to the number of files - it does not report any progress based on file sizes.

This involves a trade-off. Building the entire file list ahead of time is more memory-costly and it can significantly delay the start of the actual transfer. As you would expect, the more files there are, the longer the delay will be and the more memory it will require.

The following is from the rsync manual (source - http://rsync.samba.org/ftp/rsync/rsync.html ):

-r, --recursive

This tells rsync to copy directories recursively. See also --dirs (-d). Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

See also https://rsync.samba.org for specific version differences (scroll down and check out the Release News links).

3
  • 3
    Thanks for this, the other answers result in a %age that keeps going up and down! Nov 29, 2016 at 16:05
  • 1
    Not quite perfect yet, as the percentage is always the ratio of the current transfer to the total amount of data. So if one transfer is interrupted at 40%, the next transfer will start at 0% and finish at 60%, which is a little confusing. Nevertheless, this is a lot clearer than the accepted answer, in my opinion.
    – iFreilicht
    Oct 17, 2020 at 21:18
  • 1
    This will have a slow start but is informative and not resource hungry: nice -n 19 ionice -c 2 -n 7 rsync --archive --info=progress2 --no-inc-recursive --human-readable SRC DST
    – Pablo A
    Sep 30, 2022 at 19:50
50

You can with 'pv' (apt-get install pv with Debian and ubuntu). I recommend to monitor the number of files transferred, since the amount of data transferred is not correlated to the size of files but on the delta between the source and destination. And counting files will count the same progress for one big delta and another one with small delta. Which means that in any case the ETA estimation might be far off. The size-based ETA only works if your destination is empty, in this case delta == size of source.

The general idea is to emit one line per file 'transferred' from rsync, and count those lines with 'pv':

rsync -ai /source remote:/dest | pv -les [number of files] >/dev/null

I tend to backup whole filesystems (for several reasons), in this case you can use the much cheaper df to get the number of files (rather than du or find wich will traverse your source hierarchy another time after rsync did it). The -x option appears to make sure rsync stays on the same source filesystem (and does not follow other inner mounts):

rsync -aix /source remote:/dest | pv -les $(df -i /source | perl -ane 'print $F[2] if $F[5] =~ m:^/:') >/dev/null

If you want to count files in /source in a general way, use find /source|wc -l (warning again: might be slow and heavy on I/O).

2
  • 1
    As zerodeuz specified, use "df" only when you are rsyncing the whole partition, as df -i /source gets the number of inodes (files) from all the partition in which /source reside. Otherwise use 'find' to count files inside a directory inside the "$()" expression.
    – lepe
    Sep 17, 2015 at 1:43
  • 1
    An alternative to du or find - let's say you're finishing an incomplete copy or doing an in-place update with deletes - is to use rsync -ai --dry-run to get the same file list the main run will be passing to pv
    – Izkata
    Apr 24, 2016 at 6:07
40

danakim is correct. There are no trivial ways to add a total progress indicator.

The reason for this is that when rsync looks at a list of files to sync, it doesn't know ahead of time which files will need to change. If you are doing delta transfers, the deltas themselves have to be calculated ahead of time to give a total picture of the work that needs to be done.

In other words, the easiest way to calculate how much work there is to be done is to actually do it.

5
  • 45
    You could still have a simple indicator like (data transferred + data skipped)/(total data in source), or (# files transferred or skipped)/(# files in source). It's not going to be particularly accurate, but it'd give an idea. Good for when you're doing a big transfer at the end of the day, and you're wondering whether to wait around and turn off the computer, or let it run for the night...
    – naught101
    May 29, 2012 at 6:52
  • 3
    I don't believe that conclusion is right. I think @naught101 is being more fair to the question, and I think predicating an answer on not using --size-only or the like is further inaccurate. May 24, 2014 at 0:11
  • 3
    At the time when I wrote the reply, it was accurate -- rsync did not have a mechanism for a total progress indicator. And yes, you could write your own, but very few people would. Jun 16, 2014 at 21:11
  • 6
    Comment for newcomers: It's now possible: serverfault.com/a/441724/422003
    – imposeren
    Jun 25, 2017 at 23:09
  • using two passes, first with "--dry-run" (total expected file count) and second with detailed output logged to a file (and counting lines complete or using a pipe to 'pv'), allows easily estimating completion. The downside is scanning large nested directories twice. Depending on the need, i.e. ensuring safe customer data migrations, this might be acceptable.
    – ives
    Jan 28, 2019 at 18:52
31

For long transfers, I'm happy with running du -s on both sides. Even watch -n1 du -s, if I feel really anxious.

watch executes a command (du -s here) periodically (every 1 second here) and shows the output fullscreen.

3
  • 2
    // , Clever! According to linfo.org: "The du (i.e., disk usage) command reports the sizes of directory trees inclusive of all of their contents and the sizes of individual files. This makes it useful for tracking down space hogs, i.e., directories and files that consume large or excessive amounts of space on a hard disk drive (HDD) or other storage media." Jul 21, 2015 at 1:44
  • 11
    I don't know what kind of long transfers you do, where du is viable. du is slow as hell when the result will be in TB and M of files.
    – Nobody
    Apr 30, 2016 at 10:47
  • 3
    For my "long transfers" a du operation takes about an hour to complete, during which rsync is running much slower, because they're fighting over access to the disk. Feb 1, 2017 at 4:24
13

Basically no. You can only show progress per-file with the --progress flag, but that is about it.

I am guessing you can write a wrapper around it or use any of the patches that you already find but you must ask yourself if it is really worth it, do you actually need a total progress for rsync?

4
  • Here is a wrapper script which shows the total progress and file progress as a nice progress bar: gist.github.com/JohannesBuchner/4d61eb5a42aeaad6ce90
    – j13r
    Oct 21, 2015 at 17:47
  • 3
    You can change the way --progress work using --info=progress2, it'll show you the global progress. You will need to disable the incremental recursive algorithm for it to be useful, so --info=progress2 --no-inc-recursive
    – mat
    Mar 21, 2016 at 10:37
  • @j13r you can share this gist as an answer as well! its great option!
    – skywinder
    Feb 18, 2017 at 13:36
  • I'm using rsync to transfer 400+ GB of data from one server to another. I would like to know how much progress is being made on the total transfer. I have resorted to SSHing into the destination machine and running du -hs on the target directory every 60 seconds with watch
    – intcreator
    Feb 12, 2022 at 16:52
9

I also searched for how to show the total progress with rsync and I've found a useful answser from this post: https://stackoverflow.com/questions/7157973/monitoring-rsync-progress

Basicly, you can use --info=progress2 in the dev version of rsync 3.1.0. Here is what the doc has said:

There is also a --info=progress2 option that outputs statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify --info=name0 if you want to see how the transfer is doing without scrolling the screen with a lot of names. (You don't need to specify the --progress option in order to use --info=progress2.)

1
  • 2
    --info=name0 is gold 🌈👑
    – ipatch
    Jun 21, 2018 at 17:57
8

Use

lsof -ad3-999 -c rsync

To see what files rsync currently has open (will show filesize) rsync copies into a hidden file locally

2
  • I wanted to post this for i in $(pgrep 'rsync'); do ls -l /proc/$i/fd; done but your solution is simpler.
    – Aalex Gabi
    Aug 25, 2019 at 15:03
  • interesting approach :)
    – Victor
    Jan 31, 2021 at 0:56
8

I used the answer from zerodeux and wrote my own little bash script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable"
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

echo "Executing dry-run to see how many files must be transferred..."
TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"
8

If you don’t have the latest rsync (e.g., OS X has 2.6.9) and can’t use --info=progress2, here’s another alternative to save yourself from pages of scrolling text on progress:

rsync -aPh <source> <destination> | xargs -L1 printf "\33[2K\rTransferring: %s"

That will print out, on one line, the name of the latest file being transferred:

Transferring: the-latest.file

6

I used the answer from zerodeux and wrote my own little BASH script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable"
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

#echo "Executing dry-run to see how many files must be transferred..."
TODO=$(find ${SOURCES} | wc -l)

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

I changed the TODO dry-run to

TODO=$(find ${SOURCES} | wc -l)

It finds the number of files very fast!

2
  • find works so much better than rsync --dry-run!
    – hopeseekr
    Jan 5, 2013 at 2:11
  • 4
    find only works if you're rsync-ing locally. rsync --dry-run works with remote sources too...
    – voretaq7
    May 8, 2013 at 17:29
4

Note the caveat here that even --info=progress2 is not entirely reliable since this is percentage is based on the number of files rsync 'knows' about at the time when the progress is being displayed. This is not necessarily the total number of files that needed to be sync'd (for instance, if it discovers a large number of large files in a deeply nested directory). One way to ensure that --info=progress2 doesn't 'jump back' in the progress indication would be to force rsync to scan all the directories recursively before starting the sync (instead of its default behavior of doing an incrementally recursive scan), by also providing the --no-inc-recursive option. Note however that this option will also increase rsync memory usage and run-time.

1
  • 3
    Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias. (See other answer)
    – koppor
    Jan 13, 2019 at 8:17
4

I'd make this a comment but don't have enough reputation. In response to naught101's comment to the chosen answer, the --progress option shows how many files have been transfered out of the total amount to transfer. I didn't realize this until looking at this post and looking at the output more carefully.

The 'to-check' stat shows how many files are left out of the total. This is of most use when rsync'ing to a new destination so you know all files will be fully copied.

From the man page:

When [each] file transfer  finishes,  rsync  replaces  the
progress line with a summary line that looks like this:

    1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

In this example, the file was  1238099  bytes  long  in
total,  the average rate of transfer for the whole file
was 146.38 kilobytes per second over the 8 seconds that
it took to complete, it was the 5th transfer of a regu-
lar file during the current rsync  session,  and  there
are 169 more files for the receiver to check (to see if
they are up-to-date or not) remaining out  of  the  396
total files in the file-list.
4

I made a progress bar for rsync in a wrapper. With --info=progress2 from new versions of rsync no patching is necessary.

Usage - replace rsync with rsyncy:

rsyncy -a FROM/ TO

rsyncy

More info on GitHub, install with pip3 install --user rsyncy

3

I use a script that extracts the information from /proc//io for the rsync process (or any other process for that matter) and knowing the total amount to be transferred is calculates the progress.

#!/bin/bash

usage()
{
   echo "usage: $0 PID BASEMSIZE [DELAY[s|m|h]]"
}

if [ $# -lt 2 ]; then
   usage
   exit 1
elif [ $# -eq 3 ]; then
   DELAY=$3
else
   DELAY=5s
fi

PID=$1
PBASE=`echo "scale=2; $2/1024"|bc`

R_PID=$PID
W_PID=$PID

R_SPEED_MAX=0
W_SPEED_MAX=0
R_SPEED_CUM=0
W_SPEED_CUM=0
R_SPEED_AVG=0
W_SPEED_AVG=0

ETA=0
ETA_H=0
ETA_M=0
ETA_S=0

while [ ! -r /proc/$PID/io ];
do
   clear
   echo "Waiting for process with PID=$PID to appear!"
   sleep 1
done

B_READ_PREV=`cat /proc/$R_PID/io|awk '$1 ~ /^read_bytes/ {print $2}'`
B_WRITE_PREV=`cat /proc/$W_PID/io|awk '$1 ~ /^write_bytes/ {print $2}'`
T1=`date +%s.%N`

count=0
while true
do
   [ ! -r /proc/$PID/io ] && break
   clear
   B_READ=`cat /proc/$R_PID/io|awk '$1 ~ /^read_bytes/ {print $2}'`
   B_WRITE=`cat /proc/$W_PID/io|awk '$1 ~ /^write_bytes/ {print $2}'`
   BL_READ=`echo "scale=2; ($B_READ-$B_READ_PREV)/1048576"|bc`
   BL_WRITE=`echo "scale=2; ($B_WRITE-$B_WRITE_PREV)/1048576"|bc`
   GB_DONE=`echo "scale=2; $B_WRITE/1073741824"|bc`
   PDONE=`echo "scale=2; $GB_DONE*100/$PBASE"|bc`
   T2=`date +%s.%N`
   TLOOP=`echo "scale=2; ($T2-$T1)/1"|bc`
   R_SPEED=`echo "scale=2; $BL_READ/$TLOOP"|bc`
   W_SPEED=`echo "scale=2; $BL_WRITE/$TLOOP"|bc`

   if [ $count -ge 1 ]; then
      R_SPEED_CUM=`echo "scale=2; $R_SPEED_CUM+$R_SPEED"|bc`
      R_SPEED_AVG=`echo "scale=2; $R_SPEED_CUM/$count"|bc`
      W_SPEED_CUM=`echo "scale=2; $W_SPEED_CUM+$W_SPEED"|bc`
      W_SPEED_AVG=`echo "scale=2; $W_SPEED_CUM/$count"|bc`
      [ `echo "scale=2; $W_SPEED > $W_SPEED_MAX"|bc` -eq 1 ] && W_SPEED_MAX=$W_SPEED
      [ `echo "scale=2; $R_SPEED > $R_SPEED_MAX"|bc` -eq 1 ] && R_SPEED_MAX=$R_SPEED
   fi

   if [ `echo "scale=2; $W_SPEED_AVG > 0"|bc` -eq 1 ]; then
      ETA=`echo "scale=2; (($PBASE-$GB_DONE)*1024)/$W_SPEED_AVG"|bc`
      ETA_H=`echo "scale=0; $ETA/3600"|bc`
      ETA_M=`echo "scale=0; ($ETA%3600)/60"|bc`
      ETA_S=`echo "scale=0; ($ETA%3600)%60"|bc`
   fi

   echo "Monitoring PID: $PID"
   echo
   echo "Read:       $BL_READ MiB in $TLOOP s"
   echo "Write:      $BL_WRITE MiB in $TLOOP s"
   echo
   echo "Read Rate:  $R_SPEED MiB/s ( Avg: $R_SPEED_AVG, Max: $R_SPEED_MAX )"
   echo "Write Rate: $W_SPEED MiB/s ( Avg: $W_SPEED_AVG, Max: $W_SPEED_MAX )"
   echo
   echo "Done: $GB_DONE GiB / $PBASE GiB ($PDONE %)"
   [ `echo "scale=2; $ETA > 0"|bc` -eq 1 ] && printf "ETA: %02d:%02d:%05.2f (%.2fs)\n" $ETA_H $ETA_M $ETA_S $ETA
   echo "Elapsed: `ps -p $PID -o etime=`"

   T1=`date +%s.%N`
   sleep $DELAY
   B_READ_PREV=$B_READ
   B_WRITE_PREV=$B_WRITE
   ((count++))
done
echo "----- Finished -------------------------------------------------------------------"
3

If your version of rsync does not accept the --info=progress2 option, you can use tqdm:

To install:

pip install tqdm

To use:

$ rsync -av /source /dest | tqdm --unit_scale | wc -l
10.0Mit [00:02, 3.58Mit/s]
1

Perhaps you can combine pv with rsync. Especially the parameter --size could by helpful. Taking a look at the docs, something like pv --size $(du -sb . | awk '{print $1}') | rsync -av . host:/your/path should work.

Here you'll find the docs and software.

Haven't tried this on my own.

1

Might be a bit late here but future answer-seekers might benefit.

This was bugging me too so I thought I'd get down and dirty and write my first script. The package zenity must be installed (sudo apt-get install zenity), but I'm sure it will probably be there already. Also, I use wmctrl (window manager control) to change the title of the progress dialog when it's done, it's easily installable but won't make a difference if you don't. I just like to see when it's done in my panel.

The script basically asks for a source and destination directory, calculates the percentage of the destination over the source in size using du and displays a progress bar.

Note: This only works for complete directory/file syncing (I usually use it to make backups of apt cache), so no --exclude=/file/in/Source-directory option. Also won't work if there are files/directories in Destination directory not in the source directory. I'm not sure if it works for remote sources/destinations since never I've had the need for it or the resources to test it.

PS. This script might be very badly written or very inefficient, (script-virgin here), but at least it serves it's purpose and of course you're welcome to edit and improve it to suit your needs. PSS. Also, couldn't get the cancel button to kill rsync so I just removed it.

    #!/bin/bash
set -e;

WELC="Running RsyncP as $USER";

function echo_progress()
{
    while (($TRANSFER_SIZE > 1000));    
    do  
        DEST_SIZE=$(du -s $DEST_FOLDER | cut -d / -f 1); 
        ((TRANSFER_SIZE=$SOURCE_SIZE-DEST_SIZE)); 
        PROGRESS_PERC=$((DEST_SIZE*100/SOURCE_SIZE));
        echo $PROGRESS_PERC;
        sleep 0.1s;
    done;
    echo 100;
    zenity --info --width=250 --title=RsyncP --text="File syncing complete!";
}

function get_input()
{
    dirs=$(zenity --forms --width=500 --title="RsyncP" --text="Enter source And destination directories" --add-entry="Source: " --add-entry="Destination: " --separator=" ");

    SOURCE_FOLDER=$(echo $dirs | cut -d' ' -f 1);
    DEST_FOLDER=$(echo $dirs | cut -d' ' -f 2);

    OPTIONS=-$(zenity --list --title="RsyncP Options" --text="Select rsync options" --separator='' --height=470 --width=470 --checklist --column "activate" --column "Option" --column "Description" FALSE v "Verbose (Terminal only)" FALSE q "Quiet, supress non-error messages (Terminal only)" FALSE P "Progress (Terminal only)" FALSE a "Archive (lrpog)" TRUE r "Recurse into directories" FALSE p "Preserve permissions" FALSE o "Preserve owner" FALSE g "Preserve group" FALSE l "Copy symlinks as symlinks");

    zenity --question --no-wrap --title="RsyncP" --width=500 --text="rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER\nDo you want to continue?";

    SOURCE_SIZE=$(du -s $SOURCE_FOLDER | cut -d / -f 1); 
    DEST_SIZE=$(du -s $DEST_FOLDER | cut -d / -f 1); 
    PROGRESS_PERC=$((DEST_SIZE*100/SOURCE_SIZE)); 
    TRANSFER_SIZE=1001;
}

if [ "$(id -u)" != "0" ]; then
    zenity --question --title=RsyncP --text="$WELC, Continue?";
    get_input;
    rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER & 
    echo_progress | zenity --progress --title=RsyncP --no-cancel --auto-close --text="Copying from \n$SOURCE_FOLDER to \n$DEST_FOLDER" ;

else            
    zenity --question --title=RsyncP --text="$WELC, Continue?";
    get_input; 
    sudo rsync  $OPTIONS $SOURCE_FOLDER $DEST_FOLDER & 
    echo_progress | zenity --progress --title=RsyncP --no-cancel --auto-close --text="Copying from \n$SOURCE_FOLDER to \n$DEST_FOLDER" ;
fi
1
  • Take a look at YAD. It's a fork(rewrite?) of Zenity and is like Zenity... but on steroids. My scripts use it for everything with fallback to Zenity since I have a machine with only Zenity on it. The command options are similar enough for that but you just loose a lot of niceties with Zenity. Mar 10, 2018 at 16:26
0

If you don’t have the latest rsync (e.g., OS X has 2.6.9) and can’t upgrade with brew upgrade rsync , you don't .use --info=progress2, or another alternativeyou only can see files output without addicional commands:

rsync -avh

0

--info=progress2 works well

if you need to add data, then to my view it is convenient to first display the maximum percentage

#!/bin/bash

SOURCE=a
TARGET=b
TMP_F=/tmp/rsync.log
PARAMS="--info=progress2 --stats -r -tgo -p -l --update -H $SOURCE $TARGET"

display_total_copy_persent() {
    rsync --dry-run $PARAMS > $TMP_F
    total=`grep "^Total file size" $TMP_F | awk '{print $4}' | tr -d ','`
    trans=`grep "^Total transferred file size" $TMP_F | awk '{print $5}' | tr -d ','`
    if [ $total != 0 ]; then persent=$(($trans/($total/100))); fi
    printf "\033[0;32m[##################################################]\033[0m\n"
    printf "\033[0;32m[ Will be copied = \033[0;34m$persent%%\033[0;30m ]\n"
    printf "\033[0;32m[##################################################]\033[0m\n"
}

start_rsync_progress() {
    rsync $PARAMS
}

display_total_copy_persent
start_rsync_progress

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .