Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a way to get all website addresses that have not been blocked by firewall for the last week sorting by IP address using squid logs?

Here's an example of the log file

1250388539.781    388 TCP_MISS/200 1931 GET - DIRECT/ application/
1250388540.180    382 TCP_MISS/200 923 GET - DIRECT/ application/
1250388549.554    657 TCP_MISS/200 430 GET - DIRECT/ text/plain
1250388583.580    675 TCP_MISS/200 1579 GET - DIRECT/ text/plain
1250388588.951    702 TCP_MISS/405 1368 LOCK - DIRECT/ text/html
1250388595.574    674 TCP_MISS/200 1579 GET - DIRECT/ text/plain
1250388703.596    667 TCP_MISS/200 1579 GET - DIRECT/ text/plain
1250388746.959    699 TCP_MISS/405 1368 LOCK - DIRECT/ text/html
1250388761.595    676 TCP_MISS/200 1578 GET - DIRECT/ text/plain
1250388772.590    678 TCP_MISS/200 1580 GET - DIRECT/ text/plain
1250388803.588    663 TCP_MISS/200 620 GET - DIRECT/ text/plain
1250388835.578    665 TCP_MISS/200 430 GET - DIRECT/ text/plain
1250388835.910    678 TCP_MISS/200 1578 GET - DIRECT/ text/plain
1250388848.523    408 TCP_MISS/200 663 POST - DIRECT/ application/

System Specs: Centos5

I need to determine who visited which site for a week.

Is this possible?

share|improve this question
You'll need to provide a little more detail of your setup for someone to answer this. – David Pashley Oct 8 '09 at 14:35
up vote 2 down vote accepted

Yes, you can put this together with basic unix commands like awk, grep and sort. Or you could hack something together in perl. Or you could load it into a database and do all sorts of mining and reporting.

cat log | grep "TCP_MISS/200" | awk '{ print $3, $7 }' | sort | uniq

Note that matching physical people with proxy or web access logs is illegal in many countries and may require a court order. Your profile suggests you are in South Africa, in which case there is a constitutional right to not have "the privacy of their communications infringed". (South African Constitution of 1996, Section 14)

share|improve this answer
You don't need grep command when awk is there: sudo cat /var/log/squid/access.log | awk '/TCP_MISS\/200/{ print $3, $7 }' | sort | uniq – mezgani Oct 9 '09 at 9:26

You can use this command to browse connectivity in real time, methods and bytes:

sudo tail -f /var/log/squid/access.log |awk '{printf(" %-20s | %-15s |  %-28s | %-6s | %-5s |%-s\n",strftime("%c",$1),$3,$4,$5,$6,substr($7,0,50))}'

Also, for some kind of report i use sarg, and sometimes a python script named sqview

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.