Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run this script as quanta suggest me

#!/bin/bash

EDATE=$(tail -1 a | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < a

I checked that and it seems that your script want to check log file line by line. Since it has more than 14000 items it takes a lot. So when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked! It's impossible! It just work for short log file I think. Is the problem because of while?

share|improve this question
    
It seems like this could be a lot more efficient, you're right. Could you add a few lines of the original source file and few lines of the intended output for those lines? –  SmallClanger Sep 23 '11 at 13:35
    
So for a given apache log file, all you want is a file as1 containing one unix epoch time (5 days prior to the log date) per line? Or do you want the file to contain all of the intermediate output, too? –  SmallClanger Sep 23 '11 at 14:01
    
yeah exactly file as1 contains one date and that is 5 days prior to the end date. I just need that file not all of those echo commands ! –  matarsak Sep 23 '11 at 14:11
    
No no. The aa.log should be the Apache access log. –  quanta Sep 24 '11 at 16:36
    
I checked that and it seems that your script want to check log file line by line . since it has more than 14000 items it takes alot . so when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked ! it's impossible ! it just work for short log file I think . is the problem because of while ? –  matarsak Sep 24 '11 at 17:05
show 2 more comments

2 Answers

up vote 2 down vote accepted

This one liner (I've split it for clarity) should give you the same result. You can optionally add > as1 to the end to redirect the output to a file. Put the path to the apache log file where I've put <logfile>

for d in \
 $(sed -nre 's/.*\[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' <logfile> | date +%s -f-);
do echo $[ $d - 86400 * 5]; done

The date command doesn't need an explicity UTC formatted date for the -dargument, although it doesn't understand dates as the apache logs output them, so I've done some substitution to swap the slashes and the colon separating the date and time with spaces.

share|improve this answer
    
I run that but unsuccessful ! I could not believe that just one line code could do what I wrote ! ./8: command substitution: line 4: syntax error near unexpected token |' ./8: command substitution: line 4: sed -nre 's/.*[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' <aa.log> | date +%s -f-)' –  matarsak Sep 23 '11 at 15:04
    
Drop the < and > from the aa.log bit. You just need the filename there. –  SmallClanger Sep 23 '11 at 15:14
    
I dont know what your magic code suppose to do but it gives me out put like this 1041022520 so your code convert the time to epoch time ? because I recognize that you also subtract 5 days , I dont understand –  matarsak Sep 23 '11 at 15:18
    
That looks about right, given that the log file sample you gave is from 2002. Use epochconverter.com to convert those numbers back to dates and you should see that they're 5 * 24 hours priror to the date in your log file. –  SmallClanger Sep 23 '11 at 15:27
    
It's a little bit different, my code get the end date and subtract 5 days from it and then print all of those 5 last days ! your's subtract 5 days from all of the log file ! –  matarsak Sep 23 '11 at 15:37
show 2 more comments

Here for you:

#!/bin/bash

EDATE=$(tail -1 aa.log | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < aa.log

UPDATE

The input:

213.46.27.204 - - [21/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [22/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [23/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [24/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [25/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [26/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [27/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [28/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [29/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [30/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""

The output:

$ ./five_last_days.sh 
27/Dec/2002:12:55:21
28/Dec/2002:12:55:21
29/Dec/2002:12:55:21
30/Dec/2002:12:55:21
01/Jan/2003:12:55:21
share|improve this answer
    
it does not work ! –  matarsak Sep 23 '11 at 17:33
    
Show us the output? –  quanta Sep 23 '11 at 17:36
    
the output is so long ! it print from 8th of December till the end ! while it should print from 27th ! the last item in log file is 213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" "" –  matarsak Sep 23 '11 at 17:39
    
Cut 30 last lines from your input, run my script, and edit your first post (to keep formatting) to show me the output. –  quanta Sep 24 '11 at 0:51
    
I checked the script with your data and the output was the same as yours , but in original logfile it does nothing ! actually when I run the cursor goes to next line and next line and after a long time I press control+C , maybe it try to check the whole logfile and because it has 14000 line it takes so much time ! I could not press enter for 2 hours so that the whole log file be checked ,how could I upload the original logfile here? –  matarsak Sep 24 '11 at 16:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.