Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to gather statistics on how long it takes to retrieve a web page once every couple of seconds.

I can do a

time wget --spider

(spider will not download the page, just check that they are there)

with this command I can see how long it took to run the command and the status of the page (200 OK, 404 NOT FOUND, etc)

The issue I'm having is that I need to keep track of the statistics. So if I hit a web page every couple of seconds and every so often I get a 404 I need to see those stats.

share|improve this question
up vote 1 down vote accepted

You could try something like this to get you going

Result=$((time wget --spider "$URL") 2>&1  | egrep  'real|response')
NumFields=$(echo $Result | awk '{print NF}')

#16 Fields if there was a 302 redirect and the result is in $13
#9 fields for 200,404 the result in in $6

if [ $NumFields -eq 16 ]
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $13,$NF}')
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $6,$NF}')
# Outputs YYYYMMDDHHMMSS,URL,Response,Time Taken
# 20110526180254,,302,0m1.000s
# 20110526180928,,200,0m0.225s
# 20110526181041,,404,0m0.089

echo $(date +"%Y%m%d%H%M%S"),"$URL","$Stats"

If you use >> to redirect the output to a file you can then either pull it into a spreadsheet or use grep etc to pull information from it.

share|improve this answer

What do you think about this?

curl --write-out %{time_total} --head >> ~/stats.txt

This will give you an output like:

HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Date: Wed, 25 May 2011 21:24:45 GMT
Expires: Fri, 24 Jun 2011 21:24:45 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 1; mode=block


Which you can then parse into a spreadsheet. It may, admittedly, not be the most elegant solution. You can clean it up with a grep chain (like grep -v Location: | grep -v Content-Type: | grep -v Date etc) or something more elegant. I'm curious to see what others come up with!

share|improve this answer

Check out Nagios for monitoring, or even if you don't want a full Nagios install check out the Nagios plugins, check_http will give you performance stats as well as a simple parse-able output. You could use a shell script to put this into a file which you could process later.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.