I have several servers that I manage with Chef. They all belong to one network and bandwidth among them is big in contrast to bandwidth to outer internet.
The problem is: many times there is a need to download some archive from the internet. When I run chef-client on every server I need, they all go to internet and download it from there. Which takes some time, when instead they could hit some server inside LAN first and check if this file is already there.
What I need is some proxy server like apt-cacher-ng, but for ordinary files, where they could be stored based on url or file checksum. So that every chef-client could check for files first in this cacher and then only in the internet, if there is no required file available in cache. Ideally this cacher could behave as a proxy - so that as soon as it gets a request for some url it downloads the file itself, stores it somewhere and returns it to the requestor.
The question is: can I configure apt-cacher-ng for that, how? Or are there any other proxy or cache servers that could be useful (apache, nginx)?
I need help with copnfiguration of this proxy server only, because as soon as it is ready, I will be able to change chef-clients' configurations myself.