1

I have a folder with about 1.5 million files and I need them to be copied to a secondary server as often as possible. What would you recommend because clearly standard rsync commands do not make it because just sending the incremental file list (while using rsync -Pcr options) takes about 30 minutes... and the file transfer about 10 seconds. How would you solve this issue fellow SysAdmins?

I currently use the following command with a 30 minute interval:

rsync -Pcr /var/primary/storage /var/secondary/ --log-file=/tmp/rsync.log

Note: The /var/primary/storage is a folder mounted to the secondary server via NFS

UPDATE:

I've also tried now the following command:

cp -aur /var/primary/storage /var/secondary/

But it is odd for me that the command asks me if I allow the overwriting of a destination file that has the exact same modified date as the source file. Isn't the update option supposed to watch it and copy by default if a source file is newer (or modified) than the destination file?

  • 3
    "As often as possible" sounds a lot like "instant". If yes, I'd go for something like SDS or DRBD – Lenniey Oct 24 '18 at 7:54
  • 4
    Or ZFS with send/receive if it doesn't need to be all that instant (but still a lot faster then rsync) – Sven Oct 24 '18 at 8:09
  • I use Reslio Sync for syncing files between computers, either in the same building or across the world. 1.5 million files is pretty extreme, but it's worth a shot. Just leave it running on both machines. It's not free for business – Tim Oct 24 '18 at 8:27
0

I think the best bet is actually monitoring or logging which files have changed. Even though this could be solved using a SDS, since you're using rsync I suggest you're using a normal filesystem.

Therefore, I think running inotifywait as a deamon is the best bet for you:

sudo inotifywait --daemon --outfile /tmp/fschanges --recursive /path/to/watch \
  --event modify --event move --event create --event delete --event close_write

When embedding this in a simple bash script that 1) reads the file, 2) pushes the changes and 3) clears the contents when done, you should get a pretty 'instant' solution:

#!/bin/bash
logfile="/tmp/fschanges"
path="/home/roy/Downloads"

# Stop deamon upon exit
trap "killall inotifywait" EXIT

# Init deamon
inotifywait --daemon --outfile $logfile --recursive $path \
      --event modify --event move --event create --event delete --event close_write &

while true; do
  # Loop trough file
  while read change; do
    srcpath=$(echo "$change" | cut -d' ' -f1)
    action=$(echo "$change" | cut -d' ' -f2)
    file=$(echo "$change" | cut -d' ' -f3)

    if [[ $action == *"DELETE"* ]]; then
      echo "DELETE: rsync $srcpath$file"
    elif [[ $action == *"CREATE"* ]]; then
      echo "CREATE: rsync $srcpath$file"
    elif [[ $action == *"WRITE"* ]]; then
      echo "WRITE: rsync $srcptah$file"
    fi
  done < $logfile

  # Clear the log
  > $logfile

  # Allow for cooldown
  sleep 1
done

Off-course the script above needs to be properly completed, but you get the general idea.

  • I think this is almost exactly the same what I've tried yesterday with this: github.com/splitbrain/Watcher This wasn't looking bad, however just starting up crashed the whole script because of the amount of files. I'll give the script a try (mainly because I like scripts) and I hope it will work. – Bert Oct 24 '18 at 8:46
  • Also: What if I just create a standard service from it? – Bert Oct 24 '18 at 8:48
  • Simply add the required headers (for example see watcher.py in your Github project) and it'll run as a service. Finally, inotifywait runs on kernel level, therefore I give it better chances of not crashing (although I don't know enough about Python to know how that runs). – royarisse Oct 24 '18 at 9:03
  • It didn't work, it just took too long. However, the solution was very strange, I'll write it down in a sec as an answer. – Bert Oct 24 '18 at 11:45
  • This question explains it all. I simply overlooked that it was an NFS. By omitting the -z flag (disabling compression) you should get better performance. Also, I wonder if the -x flag improves speeds… – royarisse Oct 25 '18 at 8:36
0

I've found the solution, however it is a bit complicated and needs some thinking. If you don't want to read it, then TLDR: No NFS, no issue and it is better to give than get

So I've tried to sync the files from FileServer-1 to FileServer-2. To ease up the stress on FileServer-1, I was thinking I could execute the rsync command on FileServer-2 and copy the files from FileServer-1. I've reversed it and ran the rsync command from the FileServer-1 to copy the folders incremental filelist to FileServer-2.Magic happened, it was faster. Still unacceptable, but faster.

After a while I had to unmount an NFS folder from both FileServers. By accident I ran the rsync command from FileServer-1 to copy the files from local machine to FileServer-2. Now the real miracle happened because it was done in about 2 minutes.

I've deleted about ~600MB of data from FileServer-2 and started rsync again just to see how much time does rsync need to copy all the files (that is about 10.000 files). It took less than 5 minutes !!!!

Ever since I have the folders unmounted (that have nothing to do with the files I'm trying to rsync) rsync started working like a beast.

Anyone has an explanation how come Rsync and NFS heates eachother like this? Or this is just a very unique phenomenon on my systems? Using CentOS 7, I've forgot to tell that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.