Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to rsync a folder with millions of files from a local directory to another, on a daily basis. The files themselves change very rarely.

I want to reduce the impact on the running server, e.g. CPU time or I/O, so I am using the command below

nice -n19 ionice -c3 rsync -ac --delete /foo /backup/

However, I found that it still causes a lot of CPU spikes and high I/O consumption during the execution of rsync command.

Is there any better way to complete my goal?

share|improve this question
    
Do you actually need the files themselves there, or can you leverage symlinks? –  MDMarra Feb 7 at 16:17

2 Answers 2

up vote 3 down vote accepted

Do you really need to compare checksums rather that either just the m date or even just the size of the file?

When you use -c

-c, --checksum              skip based on checksum, not mod-time & size

That means you need to read the entire contents of every file on both file systems. If you turn this off and use the mod-time and size, you only need to stat the directory. I don't think the bwlimit option will help much in this case, since it controls the rate between rsync processes, not the rate at which does I/O on the files to determine the checksum.

share|improve this answer

You could try using rsync's --bwlimit to limit the bandwith consumed by rsync during the copy operation.

For example, I do something like this to sync video from my security cameras offsite without clobbering my Internet connection too much:

rsync -avv --bwlimit=180 --delete --exclude '*.jpg'  remote.server.com:/stuff

--Scott

share|improve this answer
    
This only limits bandwidth on a per-file basis, so it works well with large files. With millions of small files, though, there will be very little effect. –  EEAA Feb 7 at 18:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.