Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am making a remote backup of my website. Whole catalog is about 70GB with about 5,000,000 of files total. Here is the command that I run on my backup server:

rsync -ah -e ssh --delete --link-dest=/backups/2013.09.06 backuper@88.245.49.132:/var/www/backups/2013.09.07

Process runs more than 48 hours and just hangs.

I've ran strace -p of rsync process on client (webserver where website is located) and saw, that process periodicly stops at select command ending with = 0 (Timeout) after some time, then continues.

open("mysite/files/1694201", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=10083, ...}) = 0
read(3, "\r\n\320\224\320\265\321\201\321\217\321\202\321\214 \320\273\320\265\321\202, \321\210\320\265\321\201\321\202\321"..., 10083) = 10083
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999998})
write(1, "\374\17\0\7", 4)              = 4
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999999})
write(1, "\320\260\320\262\320\260\320\271\321\202\320\265...\320\232\320\270\320\264\320\260\320\271\321\202\320\265 \320\274"..., 4092) = 4092
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999999})
write(1, "\374\17\0\7", 4)              = 4
select(2, NULL, [1], [1], {60, 0})      = 0 (Timeout)

Process hangs on the last line for a minute or so.

Why can this be happening? Why the process takes so long and never reaches the end? What could those 0 (Timeout) in strace mean?

Both servers run rsync 3.0.9, IO is not overloaded.

share|improve this question
    
If not IO, what about the network ? What kind of internet connection do you have ? 70GB through a normal DSL line can be very long. –  Kwaio Sep 9 '13 at 9:41
    
@Kwaio Speed between serves is about 3 MB/sec. But select() is a local operation, is it not? –  Temnovit Sep 9 '13 at 9:43
    
Nope, select applies on a file descriptor, which can be a network socket. Speed seems high enough, but there may be temporar overloads on the network. Try to run a continuous ping or mtr during this time and look for lag spikes correlating with timeouts. –  Kwaio Sep 9 '13 at 9:45
    
Have a look at kernel's open file descriptors too, that could be a culprit. –  Kwaio Sep 9 '13 at 9:48
add comment

2 Answers

What could those 0 (Timeout) in strace mean?

Go read up on the 5th parameter passed to select.

Plainly rsync (on its own) is not appropriate for the method you have chosen for backing up the files. It has to generate a hash for each of 5 million files and send that across the network just to find if anything has changed.

If it were me, I'd wrap it up in a script running on the source server which

  1. Checks the time (tstart) the previous successful sync was started

  2. Finds all files on the source which have a mtime > tstart

  3. rsync those files modified to the backup server

e.g.

#!/bin/bash

touch newrun
find /var/www -newer lastrun -exec rsync ....
rm -f lastrun
mv newrun lastrun
share|improve this answer
    
rsync per default actually does check for changes in modification timestamp or size. The checksum is only calculated when needed (or requested via the option --checksum). –  knoepfchendruecker Sep 10 '13 at 9:25
    
Yes - and then it COMPARES the timestamps with the reference set (i.e. across the network) –  symcbean Sep 10 '13 at 13:48
add comment

are you sure you have 5 billion files?

I'd rather tgz and rsync that tgz, since the initial comparison from src to dst would take forever if you have somewhat "normal" hds, no high-speed SAN or SSD.

where does your process is slow? during file-transfer or during initial src<->dst - check?(sending incremental file list ...)

I'd check IOWAIT on both ends, if possible. and, if machines have md-raid, cat /proc/mdstatus. very bad io-performance can be a result of a rebuilding raid (but very unlikely).

and i'd to a transfer with a single large file with --progress switched on during rsync-transfer to check network-speed.

debugging hints (you should test each possible bottleneck, even just to make sure: this is NOT the problem)

  • try rsync with -avzh --progress --stats
  • io-performance locally
  • network-performance
  • hd/raid-status (SMART), check for faulty hardware
share|improve this answer
    
Hum. that would rather cancel the intrest of rsync doing incremental backups... even if i have to admit that 5billion files is toomuch. Maybe it would be a better advice to split the backup in several sub-folders to have less files per process ? –  Kwaio Sep 9 '13 at 9:47
    
(I'm not the one asking the question) I was just suggesting to split the task between several rsyncprocess to limit the file tree comparison part time. and the per-process open file descriptors –  Kwaio Sep 9 '13 at 9:51
    
rsync can become very slow (not the transfer itself) with millions of files, because rsync intially checks the filelist src<->dst to decide which files/part of files to transfer. this can take a loooooong time if you have slow disks and manymanymany files. another fs like xfs might help a little. –  that guy from over there Sep 9 '13 at 9:56
    
oh, sorry, 3 extra zeroes! 5 million files, not billion –  Temnovit Sep 9 '13 at 10:05
1  
@Temnovit: how long does find ./ -type f | wc -l takes? what is your IOWAIT - value for that time? (you can check this with top or vmstat/iostat. still tyring to find out if this might be an os/fs - problem. what happens if you run this rsync with a local dst? –  that guy from over there Sep 9 '13 at 11:57
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.