The log is being used for user actions analytics. I need this for real-time processing and persistence of messages, think of Google Analytics implementation, obivously with different needs. Going over the rotated file is obviously needed (more for offline processing and making sure I didn't missed data), but it's easy to take care of.
My question is more for the real-time analytics part. I need to see the data as it's being written to the log file. Ideally, I would like to submit entries generated in the last X seconds (say 30 seconds) to a different machine (as new file) and process it there.
I've got an Apache log file, with many writes to it per second. I need a utility that "tail" on the log file, and every X seconds takes a chunk of new lines and creates a new file on a different machine. The other machine is part of the same network.
The reason I want it to send every X seconds is due to number of writes per second. I prefer to have a chunky communication calling to the other machine. Worst case, I might lose some data (say of the last X seconds) and fix it later on offline.
I'm planning to have a file listener running on the other machine to read those files and run some processing.
Do you have some tool you know of I can run to sync the changes to new files?
I've decided to use Apache Log pipe to call a groovy script that push the messages into Amazon SQS. This will decouple the need of another machine listening to the TCP port, which will help in case of downtime/version-upgrade. I'll make sure communicating with SQS will be async and chunky (will store X messages in memory before sending to SQS).