I have a server that maintains thousands of log files that are nested in thousands of directories. At the moment, I am using RSYNC to copy those log files to a web server where I parse them and make them available through a web application.
My issue is that, given the enormous volume of files and directories, RSYNC takes a very long time. I need to get the latest files several times an hour and it just isn't proving to be very efficient.
My question is, basically, is there a better way? Perhaps some way to tell the source server to send any log file that has changed instead of using RSYNC to scan the entire folder structure?
My single priority is to get log files that have changed transferred to the web server as quickly as possible, several times per hour. But RSYNC is proving not to be that helpful given the time it takes.