4

We need to cache all psd/zip files from our amazone S3 storage. The amount of the cache between 150-200 Gb (we need to store files at least 2 month). Now we use transparent proxy squid without any caching.

So my question: is it possible to configure "main" squid such way that all queries to the s3.amazonaws.com he will proxy to the parent squid. In the parent squid we need to cache all zip and psd files only from s3.amazonaws.com. If there are no such files in the cache the parent squid will be download and put "new" files in the cache.

How will squid works with files 1-2 gb? Is there any limitations?

The main proof of this setup is that all psd/zip files from amazone storage delivered to the local clients with maximal speeds from local cache. Because the same files at the same time used by different employees and we spent a lot of time to download those files by each employees

Can this issue be solved in different ways?

5
  • This is pretty much what aws.amazon.com/storagegateway is built for.
    – ceejayoz
    Jan 30, 2013 at 19:20
  • It's not free and require esx/esxi, but anyway thanks for the link
    – ALex_hha
    Jan 30, 2013 at 20:20
  • 1
    Squid fully supports parent and sibling caches. Large files are no problem. Jan 30, 2013 at 20:37
  • @davidSchwartz, why don't you provide your response as an answer so this question is no longer "unanswered"
    – drone.ah
    Feb 15, 2013 at 16:58
  • We are testing squid in test environment. I will let you know about result
    – ALex_hha
    Apr 13, 2013 at 13:49

1 Answer 1

4

We have tested our system about month and it's worked fine. if someone is interested, below I have added configuration files (main/parent squid)

main squid (192.168.100.1) configuration file

acl local_net src 192.168.0.0/16
acl parent_peer peername PARENT_PEER
acl parent_squid dst 192.168.100.50

acl FILE_TO_CACHE urlpath_regex \.(zip|iso|rar)$
acl STORAGE dstdomain storage.example.net

http_port 192.168.100.1:3128 intercept
icp_port 3130

cache_peer 192.168.100.50 parent 3128 3130 name=PARENT_PEER connect-timeout=7 proxy-only
cache_peer_access PARENT_PEER allow STORAGE FILE_TO_CACHE

# to connect to parent via internal interface
tcp_outgoing_address 192.168.100.1 parent_peer

# to properly get cache digest from parent
tcp_outgoing_address 192.168.100.1 parent_squid

# sent all other packets via ISP2
tcp_outgoing_address xxx.xxx.xxx.xxx local_net

parent squid (192.168.100.50) configuration file

acl main_squid src 192.168.100.1

acl FILE_TO_CACHE urlpath_regex -i \.(zip|iso|rar)$

cache allow FILE_TO_CACHE
cache deny all

http_access allow main_squid
http_access allow localhost
http_access deny all

icp_port 3130
icp_access allow main_squid
icp_access deny all

http_port 192.168.100.50:3128

cache_mem 12 GB
maximum_object_size_in_memory 64 MB
minimum_object_size 16384 KB
maximum_object_size 1024 MB

cache_swap_low 93
cache_swap_high 98

refresh_pattern \.iso$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.zip$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.rar$ 129600 100 129600 override-lastmod override-expire ignore-reload

cache_effective_user squid
cache_effective_group squid

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .