Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there a way to get all website addresses that have not been blocked by firewall for the last week sorting by IP address using squid logs?

Here's an example of the log file

1250388539.781    388 10.0.44.22 TCP_MISS/200 1931 GET http://safebrowsing-cache.google.com/safebrowsing/rd/goog-phish-shavar_s_45871-45875.45871-45875.: - DIRECT/74.125.6.219 application/vnd.google.safebrowsing-chunk
1250388540.180    382 10.0.44.22 TCP_MISS/200 923 GET http://safebrowsing-cache.google.com/safebrowsing/rd/goog-phish-shavar_a_63716-63720.63716-63717.63718-63720: - DIRECT/74.125.6.219 application/vnd.google.safebrowsing-chunk
1250388549.554    657 10.0.12.101 TCP_MISS/200 430 GET http://tracker.openbittorrent.com/announce? - DIRECT/192.121.86.2 text/plain
1250388583.580    675 10.0.12.101 TCP_MISS/200 1579 GET http://tracker.prq.to/announce? - DIRECT/192.121.86.8 text/plain
1250388588.951    702 10.0.12.13 TCP_MISS/405 1368 LOCK http://www.portforward.com/images/spacer.gif - DIRECT/63.168.21.164 text/html
1250388595.574    674 10.0.12.101 TCP_MISS/200 1579 GET http://denis.stalker.h3q.com/announce? - DIRECT/192.121.86.7 text/plain
1250388703.596    667 10.0.12.101 TCP_MISS/200 1579 GET http://open.tracker.thepiratebay.org/announce? - DIRECT/192.121.86.2 text/plain
1250388746.959    699 10.0.12.13 TCP_MISS/405 1368 LOCK http://www.portforward.com/images/spacer.gif - DIRECT/63.168.21.164 text/html
1250388761.595    676 10.0.12.101 TCP_MISS/200 1578 GET http://tpb.tracker.prq.to/announce? - DIRECT/192.121.86.5 text/plain
1250388772.590    678 10.0.12.101 TCP_MISS/200 1580 GET http://open.tracker.thepiratebay.org/announce? - DIRECT/192.121.86.3 text/plain
1250388803.588    663 10.0.12.101 TCP_MISS/200 620 GET http://tracker.openbittorrent.com/announce? - DIRECT/192.121.86.3 text/plain
1250388835.578    665 10.0.12.101 TCP_MISS/200 430 GET http://tracker.thepiratebay.org/announce? - DIRECT/192.121.86.4 text/plain
1250388835.910    678 10.0.12.101 TCP_MISS/200 1578 GET http://tracker4.finalgear.com/announce? - DIRECT/192.121.86.5 text/plain
1250388848.523    408 10.0.8.155 TCP_MISS/200 663 POST http://safebrowsing.clients.google.com/safebrowsing/downloads? - DIRECT/66.102.9.138 application/vnd.google.safebrowsing-update

System Specs: Centos5

I need to determine who visited which site for a week.

Is this possible?

share|improve this question
    
You'll need to provide a little more detail of your setup for someone to answer this. –  David Pashley Oct 8 '09 at 14:35

2 Answers 2

up vote 2 down vote accepted

Yes, you can put this together with basic unix commands like awk, grep and sort. Or you could hack something together in perl. Or you could load it into a database and do all sorts of mining and reporting.

cat log | grep "TCP_MISS/200" | awk '{ print $3, $7 }' | sort | uniq

Note that matching physical people with proxy or web access logs is illegal in many countries and may require a court order. Your profile suggests you are in South Africa, in which case there is a constitutional right to not have "the privacy of their communications infringed". (South African Constitution of 1996, Section 14)

share|improve this answer
    
You don't need grep command when awk is there: sudo cat /var/log/squid/access.log | awk '/TCP_MISS\/200/{ print $3, $7 }' | sort | uniq –  mezgani Oct 9 '09 at 9:26

You can use this command to browse connectivity in real time, methods and bytes:


sudo tail -f /var/log/squid/access.log |awk '{printf(" %-20s | %-15s |  %-28s | %-6s | %-5s |%-s\n",strftime("%c",$1),$3,$4,$5,$6,substr($7,0,50))}'

Also, for some kind of report i use sarg, and sometimes a python script named sqview

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.