I'm trying to get a measure of log file line writes over time, ideally a current rate and 1-, 5-, and 15-minute averages.

I could do something like this:

watch wc -l /var/log/<my_file>

.. and, well, calculate it myself. I could write a quick script. But I believe, deep in my heart, that I'm missing something obvious.. is there already something out there that does this?

  • .... and then what? Why not write a script to parse the logs on your chosen time intervals and do what you want with the data (perhaps logging it in a log file :/ ) – bmike Dec 19 '12 at 16:47
  • 3
    IMO I think this is more of just an arithmetic question than anything else. Every scripting language I know is very easily able to count the number of lines in a text file. So if you take a measurement at T, then take another measurement at T+5 minutes, then you know the rate of logfile line creation. Then take another measurement at T+10 and average those data points together, etc. – Ryan Ries Dec 19 '12 at 16:49

Logtop might be useful for you here.

Also pipe viewer.

Some other interesting ideas here too.

BEFORE=$(wc -l ${FILE}|cut -f1 -d' ')
sleep ${DELAY}
AFTER=$(wc -l ${FILE}|cut -f1 -d' ')
echo $(($AFTER - $BEFORE))

Gives you a count of number of lines added in 10s.


cat /var/log/myfile | while (( ( (index++) % 5 ) == 0 )) { sms/chat-session/whatever }

update: actual general working code:

dmesg | while true; do  <br> 
    if  [  $((  $(( index++ )) % 5 )) -eq 0 ]; then
        nc  -p srcport --send-only hostname port    or arduino ...:P

snmp might be a more useful idea under certain circumstances.


Using wc is a good solution on small files, but it breaks when you have big log files. To solve this problem, we will use a fifo to store the new data in the file and some control data.

mkfifo /tmp/line_counter

Then, we will send in that fifo one line containing a single 0 character for each line of the log file (we do not need the complete line). We need to use sed unbuffered here (-u) to keep the output realtime. This job is put in the background to keep the shell available, so we remember its pid in a file:

tail -f -n 0 your_log_file | sed -u -e 's/.*/0/' >> /tmp/line_counter &
echo $! > /tmp/line_counter_tail_sed_pid

Then, we need a kind of timer in the fifo, again in the background:

while true; do echo 1 >> /tmp/line_counter; sleep 1; done &
echo $! > /tmp/line_counter_timer

Then, the fun part, lets read the fifo with awk:

cat /tmp/line_counter | awk -W interactive '$0 == "0" {line++}
     $0 == "1" {count[time % (60*15)]=line; time++; printf "Time %6d: %6d lines read.\n", time, line}
     $0 == "1" && time > 60    {printf "%6d lines read in the last minute.\n",     count[(time-1) % (60*15)] - count[(time-1-60)   % (60*15)]}
     $0 == "1" && time > 60*5  {printf "%6d lines read in the last 5 minutes.\n",  count[(time-1) % (60*15)] - count[(time-1-60*5) % (60*15)]}
     $0 == "1" && time > 60*15 {printf "%6d lines read in the last 15 minutes.\n", count[(time-1) % (60*15)] - count[(time-1-60*15)% (60*15)]}'

The cat and -W interactive are needed to fight buffer.

When you are done, CTRL-C the awk and:

kill $(cat /tmp/line_counter_timer)
kill $(cat /tmp/line_counter_tail_sed_pid)
rm /tmp/line_counter /tmp/line_counter_timer /tmp/line_counter_tail_sed_pid

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.