Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are trying to write a script "sendemail.sh" to count the number of occurrence of a particular string in a log file "SendEmail.log" within the given interval. We have a log file. In that we are searching for a pattern "ReqInputMsgLog" and need to count the number of times it occurred in the given period for eg: from "2014-08-19 11:30" to "2014-08-19 11:34". And our script look like this:

#!/bin/sh
enterdate=$1
echo $enterdate
enddate=$2
enterdate1=`date +%s -d $enterdate +"%Y-%m-%d %H:%M"`

echo $enterdate1
enddate1=`date +%s -d $enddate +"%Y-%m-%d %H:%M"`
echo $enddate
count=0
cat SendEmail.log | grep "ReqInputMsgLog" | awk -F "[" '{print $3}' | awk -F "," '{print $1}' > /con/scripts_server/file.txt
for line in `cat /con/scripts_server/file.txt`
do
logdate=`echo $line | awk -F : '{print $1":"$2}'`
if [[ $logdate < $enddate1 ]];
        then
        count=`expr $count + 1`
        fi
done
echo $count

But when we are trying to execute the script by the below command its not showing the proper count.

./sendemail.sh "2014-08-19 11:30" "2014-08-19 11:34"

Log file is very big one. Small chunk has been posted here.

INFO [SIBJMSRAThreadPool : 5] [2014-08-19 11:18:24,471] SendEmail - 8/19/14 11:18 AM,ECCF25B0-0147-4000-E000-1B830A3C05A9,ReqInputMsgLog,SendEmail,<?xml version="1.0" encoding="UTF-8"?>
<in:sendEmailRequestMsg xmlns:in="http://EmailMed/EmailMedInterface" xmlns:ns0="wsdl.http://EmailMed/EmailMedInterface" xmlns:ns1="http://EmailMed/EmailMedInterface" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:me="wsdl.http://EmailMed/EmailMedInterface" xsi:type="me:sendEmailRequestMsg">
<in:sendEmail xmlns:xci0="http://EmailMed/EmailMedInterface">
INFO [SIBJMSRAThreadPool : 7] [2014-08-19 11:18:14,235] SendEmail - 8/19/14 11:18 AM,ECCEFDB2-0147-4000-E000-1B830A3C05A9,ReqInputMsgLog,SendEmail,<?xml version="1.0" encoding="UTF-8"?>
<in:sendEmailRequestMsg xmlns:in="http://EmailMed/EmailMedInterface" xmlns:ns0="wsdl.http://EmailMed/EmailMedInterface" xmlns:ns1="http://EmailMed/EmailMedInterface" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:me="wsdl.http://EmailMed/EmailMedInterface" xsi:type="me:sendEmailRequestMsg">
<in:sendEmail xmlns:xci0="http://EmailMed/EmailMedInterface">
INFO [SIBJMSRAThreadPool : 7] [2014-08-19 11:18:14,241] SendEmail - xmlText: <?xml version="1.0" encoding="UTF-8"?>

after awk command we will get a file "/con/scripts_server/file.txt" which looks similar like below:

2014-08-19 11:28:03
2014-08-19 11:28:06
2014-08-19 11:28:17
2014-08-19 11:28:53
2014-08-19 11:29:02
2014-08-19 11:29:47
2014-08-19 11:29:57
2014-08-19 11:30:07
2014-08-19 11:30:17
2014-08-19 11:30:19
2014-08-19 11:30:19
2014-08-19 11:30:22
2014-08-19 11:30:25
2014-08-19 11:30:25
2014-08-19 11:30:36
2014-08-19 11:30:51
2014-08-19 11:30:56
2014-08-19 11:30:59
2014-08-19 11:30:59
2014-08-19 11:31:08
2014-08-19 11:31:25
2014-08-19 11:32:19
2014-08-19 11:32:22
2014-08-19 11:32:27
2014-08-19 11:32:28
2014-08-19 11:32:41
2014-08-19 11:32:49
2014-08-19 11:32:59
2014-08-19 11:33:27
2014-08-19 11:33:41
2014-08-19 11:34:07
2014-08-19 11:34:14
2014-08-19 11:34:21
2014-08-19 11:34:25
2014-08-19 11:34:38
2014-08-19 11:34:50
2014-08-19 11:34:58
share|improve this question
    
Can you show some log file, input and expected output? –  val0x00ff Aug 20 at 8:39
1  
Seeing your /con/scripts_server/file.txt there is an easy way to achieve this: Try it on the file.txt directly awk '/2014-08-19 11:30/,/2014-08-19 11:34/{count++} END{ printf "There are %s lines\n", count}' /con/scripts_server/file.txt –  val0x00ff Aug 20 at 11:19
    
Thank.. It worked... But it is showing the incorrect value We have 30 entries from 2014-08-19 11:30 to 2014-08-19 11:34 but it is showing as 24 lines. 2014-08-19 11:30 2014-08-19 11:34 There are 24 lines Please help us to get the correct value. –  pruthvi Aug 21 at 4:52
    
I edited my answer. The "${1%:*}" removes the last two digits including the colon. So that means seconds were removed. –  val0x00ff Aug 21 at 7:16

2 Answers 2

Use the following to calculate the lines between two time variables. Put the following code in a file called countOcurrences.

#!/bin/bash

 awk "/$1/,/$2/"'{count++} END{ printf "There are %s lines\n",  count}' con/scripts_server/file.txt

Run it as follow.

./countOcurrences "2014-08-19 11:30:07" "2014-08-19 11:34:07"

If file.txt is filled with a new date/time each time a pattern match occurs, then the above will work.

share|improve this answer

First of all I encountered 2 errors trying to recreate your problem date: extra operand 11:34:14' Trydate --help' for more information. ./script.sh: line 15: 1408448098: No such file or directory

From what I understand you isolated the dates which interest you into file.txt according to user input and you want to count occurrences in there.

I coded this:

#!/bin/bash

#Start/End dates to encolse count range
startDate="2014-08-19 11:28:00"
endDate="2014-08-19 11:35:00"
#Concert these dates to seconds since Epoch
startDateEpoch=$(date --date="$startDate" +%s)
endDateEpoch=$(date --date="$endDate" +%s)

#Read file.txt count occurences
while read line
do

processingDate=$(date --date="$line" +%s)
  if [ $processingDate -lt $endDateEpoch ] && \
  [ $processingDate -gt $startDateEpoch ]; then
    echo "APOEL FC";
  fi

done < file.txt

Your file.txt as appended here contains 37 lines, so:

sysadmin@omg:/tmp$ ./script.sh | wc
     37      74     333

This looks correct for

startDate="2014-08-19 11:28:00"
endDate="2014-08-19 11:35:00"

Changing the dates to:

startDate="2014-08-19 11:28:03"
endDate="2014-08-19 11:34:58"

sysadmin@omg:/tmp$ ./script.sh  | wc
     35      70     315

returns 35 occurrences which looks correct since first and last dates should be excluded.

So, the conversion to seconds since Epoch must have had something wrong in the syntax and the < operator in your if statement was causing a small mess.

As general guidelines prefer to use -lt -gt (less than, greater than) operators for if. Also try to avoid enclosing command in backticks - Prefer to use $(command).

Happy coding.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.