Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The screen shot below is made using firebug. The firefox <-> web server communication captured happened after clicking login button. But I cannot see any redirection in my firefox. Does that mean that firefox and web server communicated only on http headers level?

I have ready script in ruby/mechanize to scrape some details from a forum. I tested/coded it using vBulletin.org forum. Then I managed to log into my test forum and happily moved to finalize my script using the right forum. But I cannot log in. My script returns only the login page which I assume is 'not successful login attempt'.

  • Is that possible that somebody would explain the screenshot below to me?

    • In particular why there is 11 lines? Why not just one?
    • If I can understand only kind of blindly/without thinking replies to the web server. Am I right?
  • firefox sends in the post when I click login buttonauth_username=myusername&auth_password=mypassword&auth_login=Login but my script sendsauth_username=radek&auth_password=mypassword is that ok or the &auth_login=Login part must be presented?

  • Is there any chance to simulate such communication let's say in ruby?

  • or I was thinking if I can record the communication and then replay in my script (if possible)

communication screenshot

html of the login page

<form class="login" method="post"> <fieldset>
<legend>Members Login</legend> 

<div>
<label for="auth_username">Username</label> <input id="auth_username" name="auth_username">
</div>

<div>
<label for="auth_password">Password</label> <input id="auth_password" name="auth_password" type="password">
</div>

</fieldset>
<div class="buttons">
<input name="auth_login" type="submit" value="Login"><p class="note"><a href="/forgotpassword">Forgot your password?</a></p>

</div>

</form>

my ruby script

require 'rubygems'
require 'mechanize'
#agent = WWW::Mechanize.new

agent = WWW::Mechanize.new 

page = agent.get("http://www.vbulletin.org/forum/index.php")

login_form = page.form_with(:action => 'login.php?do=login')


puts 
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }    
login_form['vb_login_username'] = 'user name'
login_form['vb_login_password'] = ''

page = agent.submit login_form

output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }

mechanize log from the try of login that doesn't work

 INFO -- : Net::HTTP::Get: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 400 bytes
DEBUG -- : Read 1424 bytes
DEBUG -- : Read 2448 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:12 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: set-cookie => PHPSESSID=7cfilg86ju2ldcgso22246hpu4; path=/, WebStats:visitorId=lSMkcwuSWEE%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/, WebStats:sessionId=%2B2HHK296t%2BQ%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
DEBUG -- : saved cookie: PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : saved cookie: WebStats:visitorId=lSMkcwuSWEE%3D
DEBUG -- : saved cookie: WebStats:sessionId=%2B2HHK296t%2BQ%3D
 INFO -- : status: 200
DEBUG -- : query: "auth_username=radek&auth_password=mypassword"
 INFO -- : Net::HTTP::Post: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: cookie => WebStats:sessionId=%2B2HHK296t%2BQ%3D; WebStats:visitorId=lSMkcwuSWEE%3D; PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : request-header: referer => http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: content-length => 43
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 650 bytes
DEBUG -- : Read 1674 bytes
DEBUG -- : Read 2698 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:13 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
 INFO -- : status: 200
share|improve this question
    
Without the above-mentioned screenshot or a better description of what's going on here we can't even begin to guess at what you are asking, much less how to solve your problem :-( –  voretaq7 Feb 1 '10 at 19:51
    
@voretaq7: I upldated my question. Hopefully it hepls. –  Radek Feb 1 '10 at 20:07
add comment

1 Answer

up vote 6 down vote accepted

Redirection is typically effected through HTTP response status codes 301 (Moved Permanently) or 302 (Found / Moved Temporarily). Those 301 and 302 HTTP response status codes used to perform redirection are included in the HTTP headers.

Just because you didn't "see any redirection in Firefox" doesn't mean redirection wasn't taking place: it was, behind the scenes.

There is another way to redirect, using JavaScript in a page. With JavaScript, the redirect can be made obvious to the user, where they can be shown a page first and then get redirected (perhaps after delay) to the other location. But, those are not HTTP redirects that travel by virtue of the HTTP header response status codes.

To address your question about Ruby: You should ask that question on Stack Overflow. But, in a nutshell, yes, you should be able to set 301 and 302 redirect response codes with any server-side web technology.


UPDATE: To address the further question in your comment below:

If you'd like to learn more about how HTTP works, then I recommend the following book: HTTP: The Definitive Guide by David Gourley & Brian Totty. You can also consult the official HTTP standard as defined by RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1.

As to why the series of requests above in your question takes 11 steps, you would need to ask the original authors of the software in question. It likely could have been implemented with less steps, but probably not only 1 step. Consider, at least one of the redirects is needed by the server to set a cookie in the user's browser before sending them on to the protected resource.

Since you're trying to replicate a custom cookie-based authentication mechanism, I also suggest you focus your studies how such systems are typically built.

If you're trying to script a login page, you should make sure what your program POSTs to the server matches a real request in every respect, including that parameter you're not currently including.

Furthermore, you need to make sure that your code is accepting and sending back any cookies sent by the server. Not preserving the cookies sent by the server is a common mistake with this kind of exercise. Additionally, you need to instruct whatever library you are using for HTTP communications to follow the redirects. Some of them don't follow redirects by default.

And finally - to debug, capture both the request and response headers and body, from the web page, and from your program, and compare them. Be suspicious of any differences.

share|improve this answer
    
@BasicallyMoney.com: To be honest I am asking on SO. With no answer so far. I believe I can code some kind of solution or workaround to that but I have no idea what is going on, so I have to understand first. HTTP communication belowns on Serverfault so I am here :-) –  Radek Feb 1 '10 at 20:09
    
@BasicallyMoney.com: so if I understood right. The web browser and web server are communicating only on http headers and cookies level. So if there is not user interaction it should be possible to script this communication, right? Is there anything you can recommend? To do? To read? Whatever? I still do not understand the concept of 11 lines or communication. –  Radek Feb 1 '10 at 20:13
    
@BasicallyMoney.com: very nice explanation. Thank you for that. One more clarification. So the browser and server can communicate only via http headers and/or cookies and send NO html body? –  Radek Feb 2 '10 at 0:06
    
@Radek: Yes, the browser and server can communicate only via HTTP headers and cookies, with no HTML body. However, it isn't a very useful conversation if, at the end, there's no HTML body. But leading up to the page that actually gets displayed, yes, there can be some back-and-forth between client and server that is only header-based: that includes HTTP redirects, HTTP authentication challenge/response, and HTTP requests to see if a cached item is still current. –  Chris W. Rea Feb 2 '10 at 15:13
    
com: thank you so much for that. Everything is much cleaer. –  Radek Feb 2 '10 at 18:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.