Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Some context:

The log is being used for user actions analytics. I need this for real-time processing and persistence of messages, think of Google Analytics implementation, obivously with different needs. Going over the rotated file is obviously needed (more for offline processing and making sure I didn't missed data), but it's easy to take care of.

My question is more for the real-time analytics part. I need to see the data as it's being written to the log file. Ideally, I would like to submit entries generated in the last X seconds (say 30 seconds) to a different machine (as new file) and process it there.


I've got an Apache log file, with many writes to it per second. I need a utility that "tail" on the log file, and every X seconds takes a chunk of new lines and creates a new file on a different machine. The other machine is part of the same network.

The reason I want it to send every X seconds is due to number of writes per second. I prefer to have a chunky communication calling to the other machine. Worst case, I might lose some data (say of the last X seconds) and fix it later on offline.

I'm planning to have a file listener running on the other machine to read those files and run some processing.

Do you have some tool you know of I can run to sync the changes to new files?


I've decided to use Apache Log pipe to call a groovy script that push the messages into Amazon SQS. This will decouple the need of another machine listening to the TCP port, which will help in case of downtime/version-upgrade. I'll make sure communicating with SQS will be async and chunky (will store X messages in memory before sending to SQS).

share|improve this question
If you need "real-time" analysis, why not pipe directly into your analysis platform (either via UDP or Unix pipes?) - makes more sense than buffering through an extraneous file. – Tomer Gabel Jul 17 '12 at 13:38
up vote 2 down vote accepted

The easiest way is to pipe the logs to another process, from which you can take the logs anywhere you want; see

logrotate is simple enough tool to do this, although you want to go with a more hardcore logging system (e.g. graylog2, logstash or splunk). For example, you can pipe your logs through netcat and send them via UDP/TCP (depending on your reliability constraints) to greylog2 for aggregation and analysis.

God's in the details, as they say :-)

share|improve this answer
Thanks Tomer. The pipe looks interesting, I can send messages to the 2nd machine via "|nc -u ..." and listening to UDP port there. How can I read from the port via nc (-l) and dump it to a file every X rows? – Oren Ellenbogen Jul 17 '12 at 12:41
You may want to pipe {{nc -ul <port>}} through logrotate, or else redirect to an output file. Be careful with netcat and UDP though, it's a little tricky to get it to work:… – Tomer Gabel Jul 17 '12 at 13:36
Yep, you can use nc -l and output that to a file, and then you can use logrotate there when a file get to a certain size (maybe 1K). So logrotate is still useful to you :-). – The Unix Janitor Jul 17 '12 at 13:38
Thanks, I've updated my question with the approach I took. – Oren Ellenbogen Jul 18 '12 at 6:37

You might find logrotate can do what you need. usually it's rotates on a daily basis or size basis, ether way you can execute a shell command on rotate, so it should be easy to a scp of your log to a remote server when rotating logs.

There's lots of example there that should get you started.

You talk about sync the files, however with logrotate you should not loose any data. You can recombine the logs at your remote end by zcat(ing) the files together. However once you find how log rotates operates, this will probably be unnecessary.

share|improve this answer
Thanks for the answer. I've added some more info to the question. logrotate will not be a good solution for my needs. – Oren Ellenbogen Jul 17 '12 at 12:42

One way is using rsync, which can use SSH for transport. It will only send the difference between the old file and the new one. In any case, you'd still have to have a program on the other end that's clever enough to remember its position in the file when the file is updated.

But since you need the file pretty much in realtime, having it sent over in 30-second chunks is going to be difficult to do in an external program. I've got a solution that's a bit complicated but may work for you:

1) If it's important to not write to disk continuously, use the BufferedLogs directive. This will store several log entries in memory and write them together to disk, rather than writing them after each request. This means that the log writing delay will happen at the source rather than at the transfer. Do note that this directive is still experimental, and it can only be set once per server so you can't change it in a virtualserver!

2) Use any program you like to put a tail on the log file and pipe it to logger. Logsurfer is one possibility, I'm sure there are others. Use a facility for it that you're not using for anything else - I'm assuming "local7" but that's up to you.

3) Have your syslog daemon on the webserver set to forward local7 to your statistics server.

4) On your statistics server, if you're using syslog-ng you can pipe the incoming logs on local7 directly to the statistics program. Or you can write to disk and set flush_timeout so it only writes every 30 seconds. In both these cases you can ignore the Apache BufferedLogs directive I mentioned above.

share|improve this answer
Thanks for the answer Jenny, please see the extra context I've added at the beginning of the question. Will running rsync on live file that Apache writes to work? – Oren Ellenbogen Jul 17 '12 at 13:05
Running rsync will work as such - but if you need the file in realtime, then I'd recommend going with syslog instead. I'll amend my answer. – Jenny D Jul 17 '12 at 13:20

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.