How do I write a script that sends a get request to a web page and returns nicely formatted http headers?
6 Answers
try this nasty hack:
wget -O - -o /dev/null --save-headers http://google.com | \
awk 'BEGIN{skip=0}{ if (($0)=="\r") {skip=1;}; if (skip==0) print $0 }'
There are quite a few modules in many different languages that will retrieve HTTP headers for you.
- Python - urllib2
- Perl - LWP::Simple
etc, etc.
curl has support to view the headers when you're downloading, or you can use the -I
option to save the headers to a file.
If you just want to view the headers.. (Not programatically) just use [live http headers][1] plugin for mozilla firefox.
-
1Now that firefox is mentioned, I just have to recommend FireBug. Perfect for analyzing what happens on a page - including network traffic. getfirebug.com May 22, 2009 at 10:40
-
This is indeed what I use now - Firebug has excellent header tools. But they are a pain in the ass to copy and paste from, Firbug is not super-stable, and you can't script it to automatically check headers on multiple files. May 22, 2009 at 15:43
This bash function will take an URL + method, or Server + path + method + port. If you use "HEAD" method, it will return the headers, if you use GET, it will return all headers and the whole reply. Supports https through openssl.
#!/bin/bash
function httpreq ()
{
if [ $# -eq 0 ]; then echo -e "httpreq SERVER PATH [GET/HEAD] [PORT]\nOR\nhttpreq URL [GET/HEAD]"; return 1; fi
if echo $1 | grep -q "://"
then
SUNUCU=$(echo $1 |cut -d '/' -f3 | cut -d':' -f1)
YOL=/$(echo $1 |cut -d '/' -f4-)
PROTO=$(echo $1 |cut -d '/' -f1)
METHOD=${2:-GET}
PORT=$(echo $1| sed -n 's&^.*://.*:\([[:digit:]]*\)/.*&\1&p')
if [ -z $PORT ]
then
if [ $PROTO == "https:" ]; then PORT=443; else PORT=80; fi
fi
else
SUNUCU=$1
YOL=$2
METHOD=${3:-GET}
PORT=${4:-80}
fi
if [ $PROTO == "https:" ];
then
echo -e "$METHOD $YOL HTTP/1.1\r\nHOST:$SUNUCU\r\n\r\n" | openssl s_client -quiet -connect $SUNUCU:$PORT
else
echo -e "$METHOD $YOL HTTP/1.1\r\nHOST:$SUNUCU\r\n\r\n" | nc $SUNUCU $PORT
fi
}