I have a server that maintains thousands of log files that are nested in thousands of directories. At the moment, I am using RSYNC to copy those log files to a web server where I parse them and make them available through a web application.

My issue is that, given the enormous volume of files and directories, RSYNC takes a very long time. I need to get the latest files several times an hour and it just isn't proving to be very efficient.

My question is, basically, is there a better way? Perhaps some way to tell the source server to send any log file that has changed instead of using RSYNC to scan the entire folder structure?

My single priority is to get log files that have changed transferred to the web server as quickly as possible, several times per hour. But RSYNC is proving not to be that helpful given the time it takes.

Thank you!

You can put your log on a LVM partition, and delta-transfer a snapshot. Blocklevel transfer should in your case faster because of no scanning overhead.

To transfer you can use

rsync --inplace 

or

xdelta 
  • Thank you for the help! Moving the log files on the source server isn't an option for us right now. But I appreciate the information and will definitely keep it in mind for the future! – Jason Feb 25 '13 at 22:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.