Using wc
is a good solution on small files, but it breaks when you have big log files. To solve this problem, we will use a fifo to store the new data in the file and some control data.
mkfifo /tmp/line_counter
Then, we will send in that fifo one line containing a single 0
character for each line of the log file (we do not need the complete line). We need to use sed
unbuffered here (-u
) to keep the output realtime. This job is put in the background to keep the shell available, so we remember its pid in a file:
tail -f -n 0 your_log_file | sed -u -e 's/.*/0/' >> /tmp/line_counter &
echo $! > /tmp/line_counter_tail_sed_pid
Then, we need a kind of timer in the fifo, again in the background:
while true; do echo 1 >> /tmp/line_counter; sleep 1; done &
echo $! > /tmp/line_counter_timer
Then, the fun part, lets read the fifo with awk
:
cat /tmp/line_counter | awk -W interactive '$0 == "0" {line++}
$0 == "1" {count[time % (60*15)]=line; time++; printf "Time %6d: %6d lines read.\n", time, line}
$0 == "1" && time > 60 {printf "%6d lines read in the last minute.\n", count[(time-1) % (60*15)] - count[(time-1-60) % (60*15)]}
$0 == "1" && time > 60*5 {printf "%6d lines read in the last 5 minutes.\n", count[(time-1) % (60*15)] - count[(time-1-60*5) % (60*15)]}
$0 == "1" && time > 60*15 {printf "%6d lines read in the last 15 minutes.\n", count[(time-1) % (60*15)] - count[(time-1-60*15)% (60*15)]}'
The cat
and -W interactive
are needed to fight buffer.
When you are done, CTRL-C the awk
and:
kill $(cat /tmp/line_counter_timer)
kill $(cat /tmp/line_counter_tail_sed_pid)
rm /tmp/line_counter /tmp/line_counter_timer /tmp/line_counter_tail_sed_pid