0

I run this script as quanta suggest me

#!/bin/bash

EDATE=$(tail -1 a | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < a

I checked that and it seems that your script want to check log file line by line. Since it has more than 14000 items it takes a lot. So when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked! It's impossible! It just work for short log file I think. Is the problem because of while?

7
  • It seems like this could be a lot more efficient, you're right. Could you add a few lines of the original source file and few lines of the intended output for those lines? Sep 23, 2011 at 13:35
  • So for a given apache log file, all you want is a file as1 containing one unix epoch time (5 days prior to the log date) per line? Or do you want the file to contain all of the intermediate output, too? Sep 23, 2011 at 14:01
  • yeah exactly file as1 contains one date and that is 5 days prior to the end date. I just need that file not all of those echo commands !
    – matarsak
    Sep 23, 2011 at 14:11
  • No no. The aa.log should be the Apache access log.
    – quanta
    Sep 24, 2011 at 16:36
  • I checked that and it seems that your script want to check log file line by line . since it has more than 14000 items it takes alot . so when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked ! it's impossible ! it just work for short log file I think . is the problem because of while ?
    – matarsak
    Sep 24, 2011 at 17:05

2 Answers 2

2

This one liner (I've split it for clarity) should give you the same result. You can optionally add > as1 to the end to redirect the output to a file. Put the path to the apache log file where I've put <logfile>

for d in \
 $(sed -nre 's/.*\[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' <logfile> | date +%s -f-);
do echo $[ $d - 86400 * 5]; done

The date command doesn't need an explicity UTC formatted date for the -dargument, although it doesn't understand dates as the apache logs output them, so I've done some substitution to swap the slashes and the colon separating the date and time with spaces.

7
  • I run that but unsuccessful ! I could not believe that just one line code could do what I wrote ! ./8: command substitution: line 4: syntax error near unexpected token |' ./8: command substitution: line 4: sed -nre 's/.*[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' <aa.log> | date +%s -f-)'
    – matarsak
    Sep 23, 2011 at 15:04
  • Drop the < and > from the aa.log bit. You just need the filename there. Sep 23, 2011 at 15:14
  • I dont know what your magic code suppose to do but it gives me out put like this 1041022520 so your code convert the time to epoch time ? because I recognize that you also subtract 5 days , I dont understand
    – matarsak
    Sep 23, 2011 at 15:18
  • That looks about right, given that the log file sample you gave is from 2002. Use epochconverter.com to convert those numbers back to dates and you should see that they're 5 * 24 hours priror to the date in your log file. Sep 23, 2011 at 15:27
  • It's a little bit different, my code get the end date and subtract 5 days from it and then print all of those 5 last days ! your's subtract 5 days from all of the log file !
    – matarsak
    Sep 23, 2011 at 15:37
0

Here for you:

#!/bin/bash

EDATE=$(tail -1 aa.log | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < aa.log

UPDATE

The input:

213.46.27.204 - - [21/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [22/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [23/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [24/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [25/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [26/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [27/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [28/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [29/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [30/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""

The output:

$ ./five_last_days.sh 
27/Dec/2002:12:55:21
28/Dec/2002:12:55:21
29/Dec/2002:12:55:21
30/Dec/2002:12:55:21
01/Jan/2003:12:55:21
5
  • it does not work !
    – matarsak
    Sep 23, 2011 at 17:33
  • Show us the output?
    – quanta
    Sep 23, 2011 at 17:36
  • the output is so long ! it print from 8th of December till the end ! while it should print from 27th ! the last item in log file is 213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
    – matarsak
    Sep 23, 2011 at 17:39
  • Cut 30 last lines from your input, run my script, and edit your first post (to keep formatting) to show me the output.
    – quanta
    Sep 24, 2011 at 0:51
  • I checked the script with your data and the output was the same as yours , but in original logfile it does nothing ! actually when I run the cursor goes to next line and next line and after a long time I press control+C , maybe it try to check the whole logfile and because it has 14000 line it takes so much time ! I could not press enter for 2 hours so that the whole log file be checked ,how could I upload the original logfile here?
    – matarsak
    Sep 24, 2011 at 16:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .