up vote 1 down vote favorite
share [g+] share [fb]

I am not a system administrator but I run a little website and today I have received a subpoena request from police, in order to comply I need to extract IP addresses from our server logs for needed time.

This is my problem:

1

I have log1.txt

/var/log/lighttpd/access.log.1.gz:84.20.132.141 180.175.44.143 - [28/Apr/2011:09:23:30 -0500] "POST /base/script.php HTTP/1.1" 200 158 "http://ref/,http://ref/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.28 Safari/530.5"
/var/log/lighttpd/access.log.1.gz:85.40.142.111 180.175.44.143 - [28/Apr/2011:09:23:30 -0500] "POST /base/script.php HTTP/1.1" 200 158 "http://ref/,http://ref/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.28 Safari/530.5"

The file contains 500+ such lines

This is the output I am expecting: 85.40.142.111, 84.20.132.141, and other ips.

2

And I have log2.txt that has exactly same format

I need to output IPs from that log as well, e.g. 85.40.142.111, 81.02.32.61,

3

I need to use output results from log1 and match them with log2 and if one IP repeats in both logs then output it. From the examples I have provided, that IP would be 85.40.142.111.

Could you assist?

link|improve this question
feedback

2 Answers

up vote 2 down vote

I'd suggest using cut, not grep:

cut -d\ -f1 log1.txt | sort | uniq > ip1.txt
cut -d\ -f1 log2.txt | sort | uniq > ip2.txt
grep -f ip2.txt ip1.txt

If the IP you're after is the second in each line, rather than the first, replace '-f1' with '-f2'.

HTH.

link|improve this answer
Thanks, but it doesn't work: cut: the delimiter must be a single character – Andrey Apr 30 '11 at 8:59
There's two spaces after the backslash; the first is the delimeter. Perhaps -d' ' instead? – caelyx Apr 30 '11 at 9:00
Thank you. Seems to work now. – Andrey Apr 30 '11 at 9:01
Thank you, but the output is not what I need. /var/log/lighttpd/access.log.1.gz:93.8.158.46. How to make it to output 93.8.158.46 instead. – Andrey Apr 30 '11 at 9:02
Oh, I gave you the wrong file format. All lines also begin with /var/log/lighttpd/access.log.*.gz:, e.g. /var/log/lighttpd/access.log.1.gz: or /var/log/lighttpd/access.log.2.gz: How to modify the cut query to work with this? Thank you! Let me also edit the original post. – Andrey Apr 30 '11 at 9:05
show 1 more comment
feedback
up vote 2 down vote

If all you want is a list of ip addresses 

#!/bin/bash
cut -d ' ' -f1 log1.txt | cut -d ':' -f2 | sort | uniq >log1.out
cut -d ' ' -f1 log2.txt | cut -d ':' -f2 | sort | uniq >log2.out
while read IP
do
    sed -n /$IP/p log2.out
done <log1.out
link|improve this answer
I gave wrong file format in the original post. I have now modified it. – Andrey Apr 30 '11 at 9:07
@Andrey: I updated my answer – Iain♦ Apr 30 '11 at 9:19
Thank you Lain and Caelyx. I have accomplished my desired objective. – Andrey Apr 30 '11 at 9:27
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.