0

Is there a way to get all website addresses that have not been blocked by firewall for the last week sorting by IP address using squid logs?

Here's an example of the log file

1250388539.781    388 10.0.44.22 TCP_MISS/200 1931 GET http://safebrowsing-cache.google.com/safebrowsing/rd/goog-phish-shavar_s_45871-45875.45871-45875.: - DIRECT/74.125.6.219 application/vnd.google.safebrowsing-chunk
1250388540.180    382 10.0.44.22 TCP_MISS/200 923 GET http://safebrowsing-cache.google.com/safebrowsing/rd/goog-phish-shavar_a_63716-63720.63716-63717.63718-63720: - DIRECT/74.125.6.219 application/vnd.google.safebrowsing-chunk
1250388549.554    657 10.0.12.101 TCP_MISS/200 430 GET http://tracker.openbittorrent.com/announce? - DIRECT/192.121.86.2 text/plain
1250388583.580    675 10.0.12.101 TCP_MISS/200 1579 GET http://tracker.prq.to/announce? - DIRECT/192.121.86.8 text/plain
1250388588.951    702 10.0.12.13 TCP_MISS/405 1368 LOCK http://www.portforward.com/images/spacer.gif - DIRECT/63.168.21.164 text/html
1250388595.574    674 10.0.12.101 TCP_MISS/200 1579 GET http://denis.stalker.h3q.com/announce? - DIRECT/192.121.86.7 text/plain
1250388703.596    667 10.0.12.101 TCP_MISS/200 1579 GET http://open.tracker.thepiratebay.org/announce? - DIRECT/192.121.86.2 text/plain
1250388746.959    699 10.0.12.13 TCP_MISS/405 1368 LOCK http://www.portforward.com/images/spacer.gif - DIRECT/63.168.21.164 text/html
1250388761.595    676 10.0.12.101 TCP_MISS/200 1578 GET http://tpb.tracker.prq.to/announce? - DIRECT/192.121.86.5 text/plain
1250388772.590    678 10.0.12.101 TCP_MISS/200 1580 GET http://open.tracker.thepiratebay.org/announce? - DIRECT/192.121.86.3 text/plain
1250388803.588    663 10.0.12.101 TCP_MISS/200 620 GET http://tracker.openbittorrent.com/announce? - DIRECT/192.121.86.3 text/plain
1250388835.578    665 10.0.12.101 TCP_MISS/200 430 GET http://tracker.thepiratebay.org/announce? - DIRECT/192.121.86.4 text/plain
1250388835.910    678 10.0.12.101 TCP_MISS/200 1578 GET http://tracker4.finalgear.com/announce? - DIRECT/192.121.86.5 text/plain
1250388848.523    408 10.0.8.155 TCP_MISS/200 663 POST http://safebrowsing.clients.google.com/safebrowsing/downloads? - DIRECT/66.102.9.138 application/vnd.google.safebrowsing-update

System Specs: Centos5

I need to determine who visited which site for a week.

Is this possible?

Improve this question
1
  • You'll need to provide a little more detail of your setup for someone to answer this.
    – David Pashley
    Oct 8, 2009 at 14:35

2 Answers 2

Reset to default
2

Yes, you can put this together with basic unix commands like awk, grep and sort. Or you could hack something together in perl. Or you could load it into a database and do all sorts of mining and reporting.

cat log | grep "TCP_MISS/200" | awk '{ print $3, $7 }' | sort | uniq

Note that matching physical people with proxy or web access logs is illegal in many countries and may require a court order. Your profile suggests you are in South Africa, in which case there is a constitutional right to not have "the privacy of their communications infringed". (South African Constitution of 1996, Section 14)

Improve this answer
1
  • You don't need grep command when awk is there: sudo cat /var/log/squid/access.log | awk '/TCP_MISS\/200/{ print $3, $7 }' | sort | uniq
    – Ali Mezgani
    Oct 9, 2009 at 9:26
0

You can use this command to browse connectivity in real time, methods and bytes: 


sudo tail -f /var/log/squid/access.log |awk '{printf(" %-20s | %-15s |  %-28s | %-6s | %-5s |%-s\n",strftime("%c",$1),$3,$4,$5,$6,substr($7,0,50))}'

Also, for some kind of report i use sarg, and sometimes a python script named sqview

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .