Squid not caching from npm, but is from curl

Question

I'm hoping I'm being blind here, but I'm trying to cache registry.npmjs.org very aggressively using squid, but really can't get it working.

I've compiled squid with --enable-http-violations, and I'm getting the warnings when it starts, and I'm using the following cache line:

refresh_pattern -i registry.npmjs.org/.* 900 20% 4320 ignore-no-store ignore-auth ignore-private ignore-no-cache ignore-reload override-expire override-lastmod

Now it seems to cache as I expect, when using curl, but not when using the npm client. I've put the two logs below but I can't see what header/difference would be causing one to cache but the other not to

Using curl (caches as expected):

1488230087.926      0 127.0.0.1 TCP_MEM_HIT/200 2710 GET http://registry.npmjs.org/should-util - HIER_NONE/- application/json [User-Agent: curl/7.51.0\r\nAccept: */*\r\nProxy-Connection: Keep-Alive\r\nHost: registry.npmjs.org\r\n] [HTTP/1.1 200 OK\r\nServer: nginx/1.10.1\r\nContent-Type: application/json\r\nLast-Modified: Mon, 08 Aug 2016 15:17:14 GMT\r\nETag: "57a8a27a-871"\r\nCache-Control: max-age=300\r\nContent-Length: 2161\r\nAccept-Ranges: bytes\r\nDate: Mon, 27 Feb 2017 20:44:44 GMT\r\nVia: 1.1 varnish\r\nAge: 0\r\nX-Served-By: cache-lcy1134-LCY\r\nX-Cache: MISS\r\nX-Cache-Hits: 0\r\nX-Timer: S1488228284.071646,VS0,VE18\r\nVary: Accept-Encoding\r\nConnection: keep-alive\r\n\r]

Using npm client (does not cache):

1488230089.133     89 127.0.0.1 TCP_MISS/304 368 GET http://registry.npmjs.org/should-util - HIER_DIRECT/151.101.16.162 - [Accept-Encoding: gzip\r\nversion: 4.1.2\r\nAccept: application/json\r\nReferer: view should-util\r\nnpm-session: 169f8528ea8fd926\r\nnpm-in-ci: false\r\nUser-Agent: npm/4.1.2 node/v7.6.0 linux x64\r\nIf-None-Match: W/"57a8a27a-871"\r\nIf-Modified-Since: Mon, 08 Aug 2016 15:17:14 GMT\r\nConnection: keep-alive\r\nHost: registry.npmjs.org\r\n] [HTTP/1.1 304 Not Modified\r\nDate: Mon, 27 Feb 2017 21:14:49 GMT\r\nVia: 1.1 varnish\r\nCache-Control: max-age=300\r\nETag: W/"57a8a27a-871"\r\nAge: 83\r\nX-Served-By: cache-lcy1132-LCY\r\nX-Cache: HIT\r\nX-Cache-Hits: 2\r\nX-Timer: S1488230089.111498,VS0,VE0\r\nVary: Accept-Encoding\r\nConnection: keep-alive\r\n\r]

Answer 1

We ran into a similar issue with using nginx proxy_cache for caching proxied objects for NPM - at speedtest.net we ended up using this project: https://github.com/mixu/npm_lazy for caching requests to NPM. It works great with private modules as well. We've used it across over 40,000 builds with no issues and it's handled npm outages without an issue.

With that said, we have run into issues with it and Yarn, though it may be Yarn at fault.