6

I'm trying to get a measure of log file line writes over time, ideally a current rate and 1-, 5-, and 15-minute averages.

I could do something like this:

watch wc -l /var/log/<my_file>

.. and, well, calculate it myself. I could write a quick script. But I believe, deep in my heart, that I'm missing something obvious.. is there already something out there that does this?

2
  • .... and then what? Why not write a script to parse the logs on your chosen time intervals and do what you want with the data (perhaps logging it in a log file :/ )
    – bmike
    Dec 19, 2012 at 16:47
  • 3
    IMO I think this is more of just an arithmetic question than anything else. Every scripting language I know is very easily able to count the number of lines in a text file. So if you take a measurement at T, then take another measurement at T+5 minutes, then you know the rate of logfile line creation. Then take another measurement at T+10 and average those data points together, etc.
    – Ryan Ries
    Dec 19, 2012 at 16:49

4 Answers 4

0

Logtop might be useful for you here.

Also pipe viewer.

Some other interesting ideas here too.

4
FILE=/var/log/syslog
DELAY=10
BEFORE=$(wc -l ${FILE}|cut -f1 -d' ')
sleep ${DELAY}
AFTER=$(wc -l ${FILE}|cut -f1 -d' ')
echo $(($AFTER - $BEFORE))

Gives you a count of number of lines added in 10s.

4

cat /var/log/myfile | while (( ( (index++) % 5 ) == 0 )) { sms/chat-session/whatever }

update: actual general working code:

dmesg | while true; do  <br> 
    if  [  $((  $(( index++ )) % 5 )) -eq 0 ]; then
        nc  -p srcport --send-only hostname port    or arduino ...:P
        sms/write/chat-session/whatever; 
    fi  
done

snmp might be a more useful idea under certain circumstances.

0
0

Using wc is a good solution on small files, but it breaks when you have big log files. To solve this problem, we will use a fifo to store the new data in the file and some control data.

mkfifo /tmp/line_counter

Then, we will send in that fifo one line containing a single 0 character for each line of the log file (we do not need the complete line). We need to use sed unbuffered here (-u) to keep the output realtime. This job is put in the background to keep the shell available, so we remember its pid in a file:

tail -f -n 0 your_log_file | sed -u -e 's/.*/0/' >> /tmp/line_counter &
echo $! > /tmp/line_counter_tail_sed_pid

Then, we need a kind of timer in the fifo, again in the background:

while true; do echo 1 >> /tmp/line_counter; sleep 1; done &
echo $! > /tmp/line_counter_timer

Then, the fun part, lets read the fifo with awk:

cat /tmp/line_counter | awk -W interactive '$0 == "0" {line++}
     $0 == "1" {count[time % (60*15)]=line; time++; printf "Time %6d: %6d lines read.\n", time, line}
     $0 == "1" && time > 60    {printf "%6d lines read in the last minute.\n",     count[(time-1) % (60*15)] - count[(time-1-60)   % (60*15)]}
     $0 == "1" && time > 60*5  {printf "%6d lines read in the last 5 minutes.\n",  count[(time-1) % (60*15)] - count[(time-1-60*5) % (60*15)]}
     $0 == "1" && time > 60*15 {printf "%6d lines read in the last 15 minutes.\n", count[(time-1) % (60*15)] - count[(time-1-60*15)% (60*15)]}'

The cat and -W interactive are needed to fight buffer.

When you are done, CTRL-C the awk and:

kill $(cat /tmp/line_counter_timer)
kill $(cat /tmp/line_counter_tail_sed_pid)
rm /tmp/line_counter /tmp/line_counter_timer /tmp/line_counter_tail_sed_pid

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .