examining http headers in shell scripts

Question

How do I write a script that sends a get request to a web page and returns nicely formatted http headers?

Answer 1

try this nasty hack:

wget -O - -o /dev/null --save-headers http://google.com | \
awk 'BEGIN{skip=0}{ if (($0)=="\r") {skip=1;}; if (skip==0) print $0 }'

Answer 2

One option might be to use curl with the --dump-header option.

Answer 3

There are quite a few modules in many different languages that will retrieve HTTP headers for you.

• Python - urllib2
• Perl - LWP::Simple

etc, etc.

Answer 4

curl has support to view the headers when you're downloading, or you can use the -I option to save the headers to a file.

Answer 5

If you just want to view the headers.. (Not programatically) just use [live http headers][1] plugin for mozilla firefox.

https://addons.mozilla.org/en-US/firefox/addon/3829

Answer 6

This bash function will take an URL + method, or Server + path + method + port. If you use "HEAD" method, it will return the headers, if you use GET, it will return all headers and the whole reply. Supports https through openssl.

#!/bin/bash
function httpreq ()
{
    if [ $# -eq 0 ]; then echo -e "httpreq SERVER PATH [GET/HEAD] [PORT]\nOR\nhttpreq URL [GET/HEAD]"; return 1; fi
    if echo $1 | grep -q "://"
    then
        SUNUCU=$(echo $1 |cut -d '/' -f3 | cut -d':' -f1)
        YOL=/$(echo $1 |cut -d '/' -f4-)
        PROTO=$(echo $1 |cut -d '/' -f1)
        METHOD=${2:-GET}
        PORT=$(echo $1| sed -n 's&^.*://.*:\([[:digit:]]*\)/.*&\1&p')
        if [ -z $PORT ]
        then
            if [ $PROTO == "https:" ]; then PORT=443; else PORT=80; fi
        fi
    else
        SUNUCU=$1
        YOL=$2
        METHOD=${3:-GET}
        PORT=${4:-80}
    fi
    if [ $PROTO == "https:" ];
    then
     echo -e "$METHOD $YOL HTTP/1.1\r\nHOST:$SUNUCU\r\n\r\n" | openssl s_client -quiet -connect $SUNUCU:$PORT
    else
     echo -e "$METHOD $YOL HTTP/1.1\r\nHOST:$SUNUCU\r\n\r\n" | nc $SUNUCU $PORT
    fi
}