3

I have several servers that I manage with Chef. They all belong to one network and bandwidth among them is big in contrast to bandwidth to outer internet.

The problem is: many times there is a need to download some archive from the internet. When I run chef-client on every server I need, they all go to internet and download it from there. Which takes some time, when instead they could hit some server inside LAN first and check if this file is already there.

What I need is some proxy server like apt-cacher-ng, but for ordinary files, where they could be stored based on url or file checksum. So that every chef-client could check for files first in this cacher and then only in the internet, if there is no required file available in cache. Ideally this cacher could behave as a proxy - so that as soon as it gets a request for some url it downloads the file itself, stores it somewhere and returns it to the requestor.

The question is: can I configure apt-cacher-ng for that, how? Or are there any other proxy or cache servers that could be useful (apache, nginx)?

I need help with copnfiguration of this proxy server only, because as soon as it is ready, I will be able to change chef-clients' configurations myself.

  • Please try to rewrite this question to avoid a direct product recommendation, otherwise it's very probably getting closed. – dawud Jun 19 '14 at 11:21
3

I think you only have to configure the http_proxy option in chef client's config file, then chef will use it for http downloads.

The proxy server itself has to be set up by yourself, of course. You could use squid, Apache mod_proxy and IIRC also ngnix to set up a caching proxy. Polipo is also a pretty neat proxy.

2

Squid is a common caching proxy you can use for this. Nginx and apache aren't really built to do the sort of forward proxying and caching you're looking for.

Basically just set it up how you like, its not too complex for a basic caching proxy, then point chef at it with http_proxy in the chef config.

  • I agree that it might not be Apache's primary use case to act as a forward proxy, but to the best of my knowledge it has this feature and should be sufficient for couple of chef clients, if an Apache is running anyways. – StephenKing Feb 11 '14 at 8:21
2

Depending on your needs, you may want to decide between an http cache (like Squid), or actually storing binaries in a local repository server.

Artifactory is an excellent repository server that can store binaries, or can be a cache for packages (like nuget, npm, gem, and so on).

Depending on how you solve this problem also depends on the design of your network, the length of time you need the cached files to be stored, and the robustness surrounding your cache.

In general, a binary repository server is a more robust method of storing files, but it also takes more infrastructure and more governance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.