0

I am making a remote backup of my website. Whole catalog is about 70GB with about 5,000,000 of files total. Here is the command that I run on my backup server:

rsync -ah -e ssh --delete --link-dest=/backups/2013.09.06 [email protected]:/var/www/backups/2013.09.07

Process runs more than 48 hours and just hangs.

I've ran strace -p of rsync process on client (webserver where website is located) and saw, that process periodicly stops at select command ending with = 0 (Timeout) after some time, then continues.

open("mysite/files/1694201", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=10083, ...}) = 0
read(3, "\r\n\320\224\320\265\321\201\321\217\321\202\321\214 \320\273\320\265\321\202, \321\210\320\265\321\201\321\202\321"..., 10083) = 10083
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999998})
write(1, "\374\17\0\7", 4)              = 4
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999999})
write(1, "\320\260\320\262\320\260\320\271\321\202\320\265...\320\232\320\270\320\264\320\260\320\271\321\202\320\265 \320\274"..., 4092) = 4092
select(2, NULL, [1], [1], {60, 0})      = 1 (out [1], left {59, 999999})
write(1, "\374\17\0\7", 4)              = 4
select(2, NULL, [1], [1], {60, 0})      = 0 (Timeout)

Process hangs on the last line for a minute or so.

Why can this be happening? Why the process takes so long and never reaches the end? What could those 0 (Timeout) in strace mean?

Both servers run rsync 3.0.9, IO is not overloaded.

4
  • If not IO, what about the network ? What kind of internet connection do you have ? 70GB through a normal DSL line can be very long.
    – mveroone
    Sep 9, 2013 at 9:41
  • @Kwaio Speed between serves is about 3 MB/sec. But select() is a local operation, is it not?
    – Temnovit
    Sep 9, 2013 at 9:43
  • Nope, select applies on a file descriptor, which can be a network socket. Speed seems high enough, but there may be temporar overloads on the network. Try to run a continuous ping or mtr during this time and look for lag spikes correlating with timeouts.
    – mveroone
    Sep 9, 2013 at 9:45
  • Have a look at kernel's open file descriptors too, that could be a culprit.
    – mveroone
    Sep 9, 2013 at 9:48

2 Answers 2

1

What could those 0 (Timeout) in strace mean?

Go read up on the 5th parameter passed to select.

Plainly rsync (on its own) is not appropriate for the method you have chosen for backing up the files. It has to generate a hash for each of 5 million files and send that across the network just to find if anything has changed.

If it were me, I'd wrap it up in a script running on the source server which

  1. Checks the time (tstart) the previous successful sync was started

  2. Finds all files on the source which have a mtime > tstart

  3. rsync those files modified to the backup server

e.g.

#!/bin/bash

touch newrun
find /var/www -newer lastrun -exec rsync ....
rm -f lastrun
mv newrun lastrun
2
  • rsync per default actually does check for changes in modification timestamp or size. The checksum is only calculated when needed (or requested via the option --checksum). Sep 10, 2013 at 9:25
  • Yes - and then it COMPARES the timestamps with the reference set (i.e. across the network)
    – symcbean
    Sep 10, 2013 at 13:48
0

are you sure you have 5 billion files?

I'd rather tgz and rsync that tgz, since the initial comparison from src to dst would take forever if you have somewhat "normal" hds, no high-speed SAN or SSD.

where does your process is slow? during file-transfer or during initial src<->dst - check?(sending incremental file list ...)

I'd check IOWAIT on both ends, if possible. and, if machines have md-raid, cat /proc/mdstatus. very bad io-performance can be a result of a rebuilding raid (but very unlikely).

and i'd to a transfer with a single large file with --progress switched on during rsync-transfer to check network-speed.

debugging hints (you should test each possible bottleneck, even just to make sure: this is NOT the problem)

  • try rsync with -avzh --progress --stats
  • io-performance locally
  • network-performance
  • hd/raid-status (SMART), check for faulty hardware
6
  • Hum. that would rather cancel the intrest of rsync doing incremental backups... even if i have to admit that 5billion files is toomuch. Maybe it would be a better advice to split the backup in several sub-folders to have less files per process ?
    – mveroone
    Sep 9, 2013 at 9:47
  • (I'm not the one asking the question) I was just suggesting to split the task between several rsyncprocess to limit the file tree comparison part time. and the per-process open file descriptors
    – mveroone
    Sep 9, 2013 at 9:51
  • rsync can become very slow (not the transfer itself) with millions of files, because rsync intially checks the filelist src<->dst to decide which files/part of files to transfer. this can take a loooooong time if you have slow disks and manymanymany files. another fs like xfs might help a little. Sep 9, 2013 at 9:56
  • oh, sorry, 3 extra zeroes! 5 million files, not billion
    – Temnovit
    Sep 9, 2013 at 10:05
  • 1
    @Temnovit: how long does find ./ -type f | wc -l takes? what is your IOWAIT - value for that time? (you can check this with top or vmstat/iostat. still tyring to find out if this might be an os/fs - problem. what happens if you run this rsync with a local dst? Sep 9, 2013 at 11:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .