Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I need to gather statistics on how long it takes to retrieve a web page once every couple of seconds.

I can do a

time wget --spider http://www.google.com/index.html

(spider will not download the page, just check that they are there)

with this command I can see how long it took to run the command and the status of the page (200 OK, 404 NOT FOUND, etc)

The issue I'm having is that I need to keep track of the statistics. So if I hit a web page every couple of seconds and every so often I get a 404 I need to see those stats.

share|improve this question

3 Answers 3

up vote 1 down vote accepted

You could try something like this to get you going

#!/bin/bash
URL="http://serverfault.com/"
Result=$((time wget --spider "$URL") 2>&1  | egrep  'real|response')
NumFields=$(echo $Result | awk '{print NF}')

#16 Fields if there was a 302 redirect and the result is in $13
#9 fields for 200,404 the result in in $6

if [ $NumFields -eq 16 ]
then
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $13,$NF}')
else
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $6,$NF}')
fi
# Outputs YYYYMMDDHHMMSS,URL,Response,Time Taken
# 20110526180254,http://www.google.oom/,302,0m1.000s
# 20110526180928,http://serverfault.com/,200,0m0.225s
# 20110526181041,http://www.google.com/fred/,404,0m0.089

echo $(date +"%Y%m%d%H%M%S"),"$URL","$Stats"

If you use >> to redirect the output to a file you can then either pull it into a spreadsheet or use grep etc to pull information from it.

share|improve this answer

What do you think about this?

curl --write-out %{time_total} --head google.ca/ >> ~/stats.txt

This will give you an output like:

HTTP/1.1 301 Moved Permanently
Location: http://www.google.ca/
Content-Type: text/html; charset=UTF-8
Date: Wed, 25 May 2011 21:24:45 GMT
Expires: Fri, 24 Jun 2011 21:24:45 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 1; mode=block

0.076

Which you can then parse into a spreadsheet. It may, admittedly, not be the most elegant solution. You can clean it up with a grep chain (like grep -v Location: | grep -v Content-Type: | grep -v Date etc) or something more elegant. I'm curious to see what others come up with!

share|improve this answer

Check out Nagios for monitoring, or even if you don't want a full Nagios install check out the Nagios plugins, check_http will give you performance stats as well as a simple parse-able output. You could use a shell script to put this into a file which you could process later.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.