Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end.

My problem is that during some unstable connection - I don't know whether it's still downloading/uploading or something went wrong. Is there a way to show the progress bar for an individual file? Something similar to what wget does by default?

Thanks!

share|improve this question
1  
I've used rsync for 10+ years and I can't believe I never asked this question. –  slm Dec 24 '12 at 1:26

3 Answers 3

This might be what you're after.

% rsync -avvz --times --stats --checksum --human-readable --acls \
    --itemize-changes --progress \
    --out-format='[%t] [%i] (Last Modified: %M) (bytes: %-10l) %-100n' \
    /usr/include/glib-2.0 my-glib-copy/

The switches breakdown as follows:

  • -avvz = archive, verbose x 2, compress
  • --times = preserve modification times
  • --stats = give some file-transfer stats
  • --checksum = skip based on checksum, not mod-time & size
  • --human-readable = output numbers in a human-readable format
  • --acls = preserve ACLs (implies -p)
  • --itemize-changes = output a change-summary for all updates
  • --progress = show progress during transfer
  • --out-format='[%t] [%i] (Last Modified: %M) (bytes: %-10l) %-100n'
    • %t = current date time
    • %i = an itemized list of what is being updated
    • %M = the last-modified time of the file
    • %-10l = the length of the file in bytes (-10 is for alignment and precision)
    • %-100n = the filename (short form; trailing "/" on dir) (-100 is for alignment and precision)

NOTE: See the man pages for rsync and rsyncd.conf for full details on the above switches.

The above command produces the following transcript:

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
[2012/12/23 21:34:46] [cd+++++++++] (Last Modified: 2010/12/19-08:13:31) (bytes: 4096      ) glib-2.0/
[2012/12/23 21:34:46] [>f+++++++++] (Last Modified: 2010/09/30-15:02:30) (bytes: 1511      ) glib-2.0/glib-object.h
       1.51K 100%    0.00kB/s    0:00:00
       1.51K 100%    0.00kB/s    0:00:00 (xfer#1, to-check=181/183)
[2012/12/23 21:34:46] [>f+++++++++] (Last Modified: 2010/09/30-15:02:30) (bytes: 2910      ) glib-2.0/glib.h
       2.91K 100%    2.78MB/s    0:00:00
       2.91K 100%    2.78MB/s    0:00:00 (xfer#2, to-check=180/183)
[2012/12/23 21:34:46] [>f+++++++++] (Last Modified: 2010/09/30-15:02:31) (bytes: 3613      ) glib-2.0/gmodule.h
       3.61K 100%    3.45MB/s    0:00:00
       3.61K 100%    3.45MB/s    0:00:00 (xfer#3, to-check=179/183)
...
...
[2012/12/23 21:34:46] [>f+++++++++] (Last Modified: 2010/09/30-15:02:31) (bytes: 8431      ) glib-2.0/gobject/gvaluecollector.h
       8.43K 100%  141.96kB/s    0:00:00
       8.43K 100%  141.96kB/s    0:00:00 (xfer#178, to-check=1/183)
[2012/12/23 21:34:46] [>f+++++++++] (Last Modified: 2010/09/30-15:02:31) (bytes: 8507      ) glib-2.0/gobject/gvaluetypes.h
       8.51K 100%  143.23kB/s    0:00:00
       8.51K 100%  143.23kB/s    0:00:00 (xfer#179, to-check=0/183)
total: matches=0  hash_hits=0  false_alarms=0 data=1305506

rsync[2996] (sender) heap statistics:
  arena:         540672   (bytes from sbrk)
  ordblks:            3   (chunks not in use)
  smblks:             7
  hblks:              2   (chunks from mmap)
  hblkhd:        401408   (bytes from mmap)
  allmem:        942080   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:          592
  uordblks:      404784   (bytes used)
  fordblks:      135888   (bytes free)
  keepcost:      134240   (bytes in releasable chunk)

rsync[2999] (server receiver) heap statistics:
  arena:         286720   (bytes from sbrk)
  ordblks:            2   (chunks not in use)
  smblks:             8
  hblks:              3   (chunks from mmap)
  hblkhd:        667648   (bytes from mmap)
  allmem:        954368   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:          672
  uordblks:      174480   (bytes used)
  fordblks:      112240   (bytes free)
  keepcost:      102352   (bytes in releasable chunk)

rsync[2998] (server generator) heap statistics:
  arena:         233472   (bytes from sbrk)
  ordblks:            4   (chunks not in use)
  smblks:             6
  hblks:              2   (chunks from mmap)
  hblkhd:        401408   (bytes from mmap)
  allmem:        634880   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:          448
  uordblks:       83152   (bytes used)
  fordblks:      150320   (bytes free)
  keepcost:      131120   (bytes in releasable chunk)

Number of files: 183
Number of files transferred: 179
Total file size: 1.31M bytes
Total transferred file size: 1.31M bytes
Literal data: 1.31M bytes
Matched data: 0 bytes
File list size: 6.30K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 350.39K
Total bytes received: 3.43K

sent 350.39K bytes  received 3.43K bytes  707.64K bytes/sec
total size is 1.31M  speedup is 3.69

Progress Bar in newer versions of rsync?

I came across this SF question:

One of the answers mentioned that in newer versions of rsync there is now support for a progress bar. It looks like this:

% ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

This progress bar still doesn't show the status of individual files in the transfer, but it does show a percentage of the overall transfer which might be helpful.

share|improve this answer
    
Thanks for the answer. It's still not quite what I'm looking for. What I really need is when you download an individual file (let's say glib-2.0/glib-object.h) - it would show me a progress bar for this file only. Imagine glib-2.0/glib-object.h being 20Gb. How would you know if it's still downloading and how much time has left? –  Salmon Dec 24 '12 at 16:39

Are you using rsync in daemon mode? The default behaviour of rsync (on Debian 6) is how you've requested. However, I'm finding in the man page for rsyncd.conf that the default behaviour in daemon mode is:

transfer logging
This parameter enables per-file logging of downloads and uploads in a format somewhat similar to that used by ftp daemons. The daemon always logs the transfer at the end, so if a transfer is aborted, no mention will be made in the log file.
If you want to customize the log lines, see the "log format" parameter.

So if using rsync in daemon mode, it looks like you'll have to add the line "transfer logging" to rsyncd.conf.

share|improve this answer
    
No, I'm not in the deamon mode and I'm not seeing the progress bar for each file, just the summary for the whole process –  Salmon Dec 24 '12 at 16:35

I think Pipe Viewer is what you need: https://code.google.com/p/pipeviewer/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.