only http header communication? - please explain

Question

The screen shot below is made using firebug. The firefox <-> web server communication captured happened after clicking login button. But I cannot see any redirection in my firefox. Does that mean that firefox and web server communicated only on http headers level?

I have ready script in ruby/mechanize to scrape some details from a forum. I tested/coded it using vBulletin.org forum. Then I managed to log into my test forum and happily moved to finalize my script using the right forum. But I cannot log in. My script returns only the login page which I assume is 'not successful login attempt'.

• Is that possible that somebody would explain the screenshot below to me?

  • In particular why there is 11 lines? Why not just one?
  • If I can understand only kind of blindly/without thinking replies to the web server. Am I right?

• firefox sends in the post when I click login buttonauth_username=myusername&auth_password=mypassword&auth_login=Login but my script sendsauth_username=radek&auth_password=mypassword is that ok or the &auth_login=Login part must be presented?

• Is there any chance to simulate such communication let's say in ruby?

• or I was thinking if I can record the communication and then replay in my script (if possible)

html of the login page

<form class="login" method="post"> <fieldset>
<legend>Members Login</legend> 

<div>
<label for="auth_username">Username</label> <input id="auth_username" name="auth_username">
</div>

<div>
<label for="auth_password">Password</label> <input id="auth_password" name="auth_password" type="password">
</div>

</fieldset>
<div class="buttons">
<input name="auth_login" type="submit" value="Login"><p class="note"><a href="/forgotpassword">Forgot your password?</a></p>

</div>

</form>

my ruby script

require 'rubygems'
require 'mechanize'
#agent = WWW::Mechanize.new

agent = WWW::Mechanize.new 

page = agent.get("http://www.vbulletin.org/forum/index.php")

login_form = page.form_with(:action => 'login.php?do=login')


puts 
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }    
login_form['vb_login_username'] = 'user name'
login_form['vb_login_password'] = ''

page = agent.submit login_form

output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }

mechanize log from the try of login that doesn't work

 INFO -- : Net::HTTP::Get: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 400 bytes
DEBUG -- : Read 1424 bytes
DEBUG -- : Read 2448 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:12 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: set-cookie => PHPSESSID=7cfilg86ju2ldcgso22246hpu4; path=/, WebStats:visitorId=lSMkcwuSWEE%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/, WebStats:sessionId=%2B2HHK296t%2BQ%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
DEBUG -- : saved cookie: PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : saved cookie: WebStats:visitorId=lSMkcwuSWEE%3D
DEBUG -- : saved cookie: WebStats:sessionId=%2B2HHK296t%2BQ%3D
 INFO -- : status: 200
DEBUG -- : query: "auth_username=radek&auth_password=mypassword"
 INFO -- : Net::HTTP::Post: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: cookie => WebStats:sessionId=%2B2HHK296t%2BQ%3D; WebStats:visitorId=lSMkcwuSWEE%3D; PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : request-header: referer => http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: content-length => 43
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 650 bytes
DEBUG -- : Read 1674 bytes
DEBUG -- : Read 2698 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:13 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
 INFO -- : status: 200

Answer 1

Redirection is typically effected through HTTP response status codes 301 (Moved Permanently) or 302 (Found / Moved Temporarily). Those 301 and 302 HTTP response status codes used to perform redirection are included in the HTTP headers.

Just because you didn't "see any redirection in Firefox" doesn't mean redirection wasn't taking place: it was, behind the scenes.

There is another way to redirect, using JavaScript in a page. With JavaScript, the redirect can be made obvious to the user, where they can be shown a page first and then get redirected (perhaps after delay) to the other location. But, those are not HTTP redirects that travel by virtue of the HTTP header response status codes.

To address your question about Ruby: You should ask that question on Stack Overflow. But, in a nutshell, yes, you should be able to set 301 and 302 redirect response codes with any server-side web technology.

---

UPDATE: To address the further question in your comment below:

If you'd like to learn more about how HTTP works, then I recommend the following book: HTTP: The Definitive Guide by David Gourley & Brian Totty. You can also consult the official HTTP standard as defined by RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1.

As to why the series of requests above in your question takes 11 steps, you would need to ask the original authors of the software in question. It likely could have been implemented with less steps, but probably not only 1 step. Consider, at least one of the redirects is needed by the server to set a cookie in the user's browser before sending them on to the protected resource.

Since you're trying to replicate a custom cookie-based authentication mechanism, I also suggest you focus your studies how such systems are typically built.

If you're trying to script a login page, you should make sure what your program POSTs to the server matches a real request in every respect, including that parameter you're not currently including.

Furthermore, you need to make sure that your code is accepting and sending back any cookies sent by the server. Not preserving the cookies sent by the server is a common mistake with this kind of exercise. Additionally, you need to instruct whatever library you are using for HTTP communications to follow the redirects. Some of them don't follow redirects by default.

And finally - to debug, capture both the request and response headers and body, from the web page, and from your program, and compare them. Be suspicious of any differences.