Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a proxy server (preferably Squid) or similar software that will let me queue say 100 URLs upfront from a list or API and cache the HTML data transparently (making parallel requests).

Then later when I request one of the URL, it gives a cached version quickly?

share|improve this question

With any caching proxy you like, you can script the cache warm-up requests - the proxy will take care of caching the responses according to its policy (make sure to allocate enough storage and set a sufficiently durable retention).

Some script in the spirit of the following example will do just fine for getting a list of URL through the proxy:

http_proxy=; export http_proxy
for my_url in `cat one_url_per_line_file` do
  wget -r -nd --delete-after $my_url
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.