Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

For research I am looking to get the actual raw output that apache sends when responding to a request, including all get data, headers, and the html. How would I go about this? I have tried:

wget --output-document=response.txt -S "http://thewebsiteinquestion.com"

This shows me the response headers all layered out nicely and sends the HTML to the response.txt, but I really want to see raw text and how this is formatted.

share|improve this question
    
Try a packet capture program, like Wireshark or tcpdump. –  joeqwerty Sep 20 '12 at 17:10
add comment

2 Answers

up vote 3 down vote accepted

I imagine it could be done with wget but I know how to do this with curl. yum install curl or whatever for your OS if you don't already have it installed:

curl -vv http://thewebsiteinquestion.com

The output will show you the raw response from the server, including the headers and body, like you wanted. This is purely a preference thing but I've generally found curl easier and more intuitive to use for testing and development than wget.

Sometimes it's also helpful to use telnet and speak raw HTTP. As a sysadm/dev I find this invaluable. For example:

$ telnet google.com 80
Trying 74.125.239.0...
Connected to google.com.
Escape character is '^]'.
GET / HTTP/1.0
Host: google.com

HTTP/1.0 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Thu, 20 Sep 2012 22:31:32 
... the rest ommitted

All you need to type by hand is the GET / HTTP/1.0 and Host: google.com lines. Then hit enter twice. This way you really see the full response from the remote HTTP server, with nothing getting in the way. As you play around with it you learn better how HTTP works, and will never get confused when redirects are happening (as you can see is happening above, via the 301 HTTP status and Location header), and your tool (browser, wget, curl) is handling them automatically for you.

Edit: how to pass GET parameters using telnet. Just add them to the path you're GETting when you speak HTTP:

$ telnet thewebsite.co.uk 80
Trying 95.131.67.221...
Connected to thewebsite.co.uk.
Escape character is '^]'.
GET /?foo=bar HTTP/1.0
Host: thewebsite.co.uk 

HTTP/1.1 200 OK
Date: Fri, 21 Sep 2012 17:43:27 GMT
Server: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
Last-Modified: Mon, 01 Jun 2009 20:10
...

To do a POST query, you have to calculate the length of the submitted data and put it in Content-Length, and that looks like:

$ telnet www.yahoo.com 80
Trying 72.30.38.140...
Connected to ds-any-fp3-real.wa1.b.yahoo.com.
Escape character is '^]'.
POST / HTTP/1.0
User-Agent: telnet
Host: www.yahoo.com
Accept: */*
Content-Length: 8
Content-Type: application/x-www-form-urlencoded

p=foobar
HTTP/1.0 200 OK
Date: Fri, 21 Sep 2012 17:57:35 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Cache-Control: private
X-Frame-Options: SAMEORIGIN
...
share|improve this answer
    
Obviously everyone's different. I can't stand curl. –  Michael Hampton Sep 20 '12 at 22:42
    
Excellent! Thank you! Brilliant way of seeing how a browser works, you're right –  beingalex Sep 21 '12 at 11:02
    
Just another quick one: telnet thewebsite.co.uk?foo=bar 80 breaks. How do I fire GET variables to the server? –  beingalex Sep 21 '12 at 11:11
1  
Thanks for the accept :-). I added examples on how to send GET and POST variables with telnet. –  Steve Kehlet Sep 21 '12 at 18:00
add comment

Add the --save-headers option to the wget command line.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.