I've built a system that needs to collect new apache logs from four machines every hour, merge them, process them and store them. For the task of collecting the new logs I've written a bunch of cron script that reads in the line of already sent lines and start sending only the new lines, then store the new line number. This is synchronized with a merging script using some .lock files.
Although this is working quite well I would like to substitute this system with a more reliable system with good error handling and perhaps autodetect of new log files (they are rotated every day). Is there something I could use to do this kind of task? Thank you!