Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We are a price comparison portal and crawl certain websites on a regular basis. I am looking for a method to save bandwidth by avoiding downloading the same page over and over again if the content did not change and instead make only a head request. Since the websites we care about are dynamic by nature the Last-Modified field or Content-Length are not a reliable indicator to whether a certain web page has changed or not. The response Etags field or even better the response MD5 would probably work fine. However most of the servers do not generate content MD5 for each request probably because it implies some CPU overhead for each response and would slow them down instead of saving some bandwidth and thus making them faster.

My question, is there a universally accepted method to create a HTTP request that will prompt the server to generate the Etags or the MD5 header for the response?

share|improve this question

There is no way for that. You cannot ask the server to return anything extra unless the web site provides a special API for that.

share|improve this answer

http head request?

Its return the same as GET but without body.

share|improve this answer
My point exactly, how to make the server return content md5 in the response to the head request? Please read the question .. – user46728 Oct 17 '11 at 6:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.