0

I am trying to setup squid to cache downloading of one of my programs , unfortunately I found out I am not caching anything, my logs looks like:

1414421675.540   1059 127.0.0.1 TCP_MISS/204 288 GET http://serverfault.com/posts/596890/ivc/d85b? - DIRECT/198.252.206.16 -
1414421676.548    769 127.0.0.1 TCP_MISS/200 1539 GET http://careers.stackoverflow.com/gethired/js - DIRECT/198.252.206.17 text/javascript
1414421677.226    674 127.0.0.1 TCP_MISS/204 442 GET http://careers.stackoverflow.com/ad/i/o8dlwjVTTd7-lrgGzmMcoFk-2LAw8 - DIRECT/198.252.206.17 image/gif
1414421685.217     48 127.0.0.1 TCP_MISS/200 4345 GET http://audaupdate.audatex.cz/SEARCH_TREE/ - DIRECT/93.189.233.60 text/html
1414421685.273     44 127.0.0.1 TCP_MISS/200 1407 GET http://audaupdate.audatex.cz/SEARCH_TREE/INDEX.DEF - DIRECT/93.189.233.60 text/plain
1414421689.198    553 127.0.0.1 TCP_MISS/200 16838 GET http://serverfault.com/questions/473940/caching-large-amounts-of-data - DIRECT/198.252.206.16 text/html
1414421689.276     35 127.0.0.1 TCP_MISS/200 1983 GET http://i.stack.imgur.com/F801B.jpg? - DIRECT/103.31.6.34 image/jpeg
1414421689.426     27 127.0.0.1 TCP_MISS/200 614 GET http://www.google-analytics.com/collect? - DIRECT/173.194.40.5 image/gif
1414421689.438     32 127.0.0.1 TCP_MISS/204 395 GET http://b.scorecardresearch.com/b? - DIRECT/193.247.167.139 -
1414421689.517    138 127.0.0.1 TCP_MISS/200 2615 GET http://engine.adzerk.net/ados? - DIRECT/107.20.144.218 application/javascript
1414421689.594    190 127.0.0.1 TCP_MISS/200 439 GET http://pixel.quantserve.com/pixel;r=545276160;a=p-c1rF4kxgLUzNc;fpan=0;fpa=P0-1294698880-1407227396456;ns=0;ce=1;cm=;je=1;sr=1920x1080x24;enc=n;dst=1;et=1414421689399;tzo=-60;ref=http%3A%2F%2Fserverfault.com%2Fquestions%2Fask;url=http%3A%2F%2Fserverfault.com%2Fquestions%2F473940%2Fcaching-large-amounts-of-data;ogl=type.website%2Cimage.http%3A%2F%2Fcdn%252Esstatic%252Enet%2Fserverfault%2Fimg%2Fapple-touch-icon%402%252Epng%3Fv%3D9b1f48ae296b%2Ctitle.Caching%20large%20amounts%20of%20data%2Cdescription.We%20need%20to%20cache%20all%20psd%2Fzip%20files%20from%20our%20amazone%20S3%20storage%252E%20The%20amount%20of%20th%2Curl.http%3A%2F%2Fserverfault%252Ecom%2Fquestions%2F473940%2Fcaching-large-amounts-of-data - DIRECT/95.172.94.12 image/gif
1414421689.996    214 127.0.0.1 TCP_MISS/200 1036 GET http://stackexchange.com/ads/launched-question/js? - DIRECT/198.252.206.16 text/javascript
1414421690.007    228 127.0.0.1 TCP_MISS/200 671 GET http://engine.adzerk.net/i.gif? - DIRECT/23.23.234.111 image/gif
1414421690.093    651 127.0.0.1 TCP_MISS/204 288 GET http://serverfault.com/posts/473940/ivc/d85b? - DIRECT/198.252.206.16 -
1414421690.128    119 127.0.0.1 TCP_MISS/200 671 GET http://engine.adzerk.net/i.gif? - DIRECT/23.23.234.111 image/gif
1414421690.229    153 127.0.0.1 TCP_MISS/200 3163 GET http://ad.doubleclick.net/adj/N7384.171779.STACKOVERFLOW.COM/B8285612.111984239;sz=728x90;click=http://engine.adzerk.net/r? - DIRECT/173.194.40.27 text/javascript
1414421690.320     42 127.0.0.1 TCP_MISS/200 6968 GET http://cdn.sstatic.net/stackexchange/Img/launched-ads/physics.png - DIRECT/141.101.114.59 image/png
1414421690.369    126 127.0.0.1 TCP_MISS/200 603 GET http://googleads4.g.doubleclick.net/pagead/adview? - DIRECT/173.194.40.13 text/html
1414421690.641    103 127.0.0.1 TCP_MISS/200 10047 GET http://s1.2mdn.net/viewad/3742665/SignUp-BannerAds-CodeBugs-c-728x90.gif - DIRECT/74.125.232.156 image/gif
1414421691.158    379 127.0.0.1 TCP_MISS/204 442 GET http://careers.stackoverflow.com/ad/i/o8dlwjVTTd7-lrgGzmMcoFk-2LAw8 - DIRECT/198.252.206.17 image/gif
1414421692.532    669 127.0.0.1 TCP_MISS/200 529 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421693.656  19965 127.0.0.1 TCP_MISS/200 5076 CONNECT adzerk-www.s3.amazonaws.com:443 - DIRECT/54.231.1.233 -
1414421699.259  23788 127.0.0.1 TCP_MISS/200 1560 CONNECT qa.sockets.stackexchange.com:443 - DIRECT/198.252.206.25 -
1414421700.747     91 127.0.0.1 TCP_MISS/200 677 GET http://pagead2.googlesyndication.com/activeview? - DIRECT/173.194.112.218 image/gif
1414421709.400    141 127.0.0.1 TCP_MISS/200 29779 GET http://audaupdate.audatex.cz/ - DIRECT/93.189.233.60 text/html
1414421710.792   1349 127.0.0.1 TCP_MISS/200 1858526 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0000A5/0000A5.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.121    297 127.0.0.1 TCP_MISS/200 439819 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0000R8/0000R8.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.518    370 127.0.0.1 TCP_MISS/200 508209 GET http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0JD050/0JD050.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed
1414421711.665     52 127.0.0.1 TCP_MISS/200 29779 GET http://audaupdate.audatex.cz/ - DIRECT/93.189.233.60 text/html
1414421712.270    548 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 -
1414421712.911    127 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 -
1414421713.531     83 127.0.0.1 TCP_MISS/200 3382 GET http://audaupdate.audatex.cz/LOG/347035-20141027155316-STATUS.LOG - DIRECT/93.189.233.60 text/plain
1414421715.584     42 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/TRACE/347035-20141027.ZIP - DIRECT/93.189.233.60 -
1414421717.463   1878 127.0.0.1 TCP_MISS/204 262 PUT http://audaupdate.audatex.cz/TRACE/347035-20141027.ZIP - DIRECT/93.189.233.60 -
1414421738.346    588 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421784.008    258 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421830.045    298 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421839.845     51 127.0.0.1 TCP_MISS/200 1624 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421839.849     55 127.0.0.1 TCP_MISS/200 1624 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421845.898   5052 127.0.0.1 TCP_MISS/200 2940 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421845.908   5057 127.0.0.1 TCP_MISS/200 2908 CONNECT autodiscover.audatex.cz:443 - DIRECT/86.110.226.145 -
1414421855.023 200872 127.0.0.1 TCP_MISS/200 9839 CONNECT www.gravatar.com:443 - DIRECT/68.232.35.121 -
1414421876.367    622 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421889.871 199446 127.0.0.1 TCP_MISS/200 1560 CONNECT qa.sockets.stackexchange.com:443 - DIRECT/198.252.206.25 -
1414421894.153    358 127.0.0.1 TCP_MISS/200 17472 GET http://serverfault.com/search/titles? - DIRECT/198.252.206.16 application/json
1414421894.399    229 127.0.0.1 TCP_MISS/204 306 POST http://serverfault.com/gps/event - DIRECT/198.252.206.16 -
1414421896.276    275 127.0.0.1 TCP_MISS/200 17554 GET http://serverfault.com/search/titles? - DIRECT/198.252.206.16 application/json
1414421896.510    233 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-title - DIRECT/198.252.206.16 application/json
1414421896.851    340 127.0.0.1 TCP_MISS/204 306 POST http://serverfault.com/gps/event - DIRECT/198.252.206.16 -
1414421921.868    499 127.0.0.1 TCP_MISS/200 391 POST http://serverfault.com/posts/0/editor-heartbeat/ask - DIRECT/198.252.206.16 application/json
1414421929.372 292573 127.0.0.1 TCP_MISS/200 4556 CONNECT clients1.google.com:443 - DIRECT/173.194.112.199 -
1414421941.513    334 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json
1414421943.762    235 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json
1414421949.862    476 127.0.0.1 TCP_MISS/200 392 POST http://serverfault.com/posts/validate-body - DIRECT/198.252.206.16 application/json

So it looks it logging what i did, and also log which files program download

http://audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/0JD050/0JD050.GFF.ZIP - DIRECT/93.189.233.60 application/x-zip-compressed

Anyway when I try to download same file again it is downloaded again from the original webserver instead of from cache. Same when I try to watch some video or anything, it is always downloading , never served from cache. Adding my config as well. Can anybody tell me where can be a problem?

The sanitised config is below:

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80      # http
acl Safe_ports port 21      # ftp
acl Safe_ports port 443     # https
acl Safe_ports port 70      # gopher
acl Safe_ports port 210     # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280     # http-mgmt
acl Safe_ports port 488     # gss-http
acl Safe_ports port 591     # filemaker
acl Safe_ports port 777     # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access allow localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow all
icp_access allow localnet
icp_access deny all
http_port 3128
hierarchy_stoplist cgi-bin ?
 cache_dir ufs c:/squid/var/cache 9000 16 256
 maximum_object_size 4096000000 KB
access_log c:/squid/var/logs/access.log squid
refresh_pattern ^ftp:       1440    20% 10080
refresh_pattern ^gopher:    1440    0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
refresh_pattern .       0   20% 4320
 range_offset_limit 409600000 KB
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
visible_hostname localhost 
coredump_dir c:/squid/var/cache

The full config can be found here: http://pastebin.com/yRV6VE16

  • 1
    The example you gave (audaupdate.audatex.cz/MASTER_APP/FULL/MODEL/GRAPHIC/0000/GRA/…) requires authentication. These files are typically not cached. Do you have another example of an URL that you think should be cached, but isn't? – Thomas Stinner Oct 27 '14 at 15:41
  • This is main thing i need to fix, so is there a chance how can i let squid cache files which required auth? – Andurit Oct 29 '14 at 12:31
  • 1
    refresh_pattern has an option to ignore auth: ignore-auth caches responses to requests with authorization, as if the originserver had sent ``Cache-control: public'' in the response header. Doing this VIOLATES the HTTP standard. Enabling this feature could make you liable for problems which it causes. See here: squid-cache.org/Doc/config/refresh_pattern – Thomas Stinner Oct 29 '14 at 12:37
0

Squid is not able to keep objects coming from CDNs like YouTube or URLs with '?' char. To cache videos, images, etc... from CDN URLs, you should use a plugin that'll do the job for you. You can try the SquidVideoBooster taking care 600+ websites including YouTube, DailyMotion, NetFlix, etc...

Link: https://sourceforge.net/projects/squidvideosbooster/ Compatible Squid 2.7, 3.4 and 3.5.

Download this plugin and test, you'll see if it fits your needs ;o)

Bye Fred

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.