Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server that maintains thousands of log files that are nested in thousands of directories. At the moment, I am using RSYNC to copy those log files to a web server where I parse them and make them available through a web application.

My issue is that, given the enormous volume of files and directories, RSYNC takes a very long time. I need to get the latest files several times an hour and it just isn't proving to be very efficient.

My question is, basically, is there a better way? Perhaps some way to tell the source server to send any log file that has changed instead of using RSYNC to scan the entire folder structure?

My single priority is to get log files that have changed transferred to the web server as quickly as possible, several times per hour. But RSYNC is proving not to be that helpful given the time it takes.

Thank you!

share|improve this question
add comment

1 Answer 1

You can put your log on a LVM partition, and delta-transfer a snapshot. Blocklevel transfer should in your case faster because of no scanning overhead.

To transfer you can use

rsync --inplace 

or

xdelta 
share|improve this answer
    
Thank you for the help! Moving the log files on the source server isn't an option for us right now. But I appreciate the information and will definitely keep it in mind for the future! –  Jason Feb 25 '13 at 22:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.