0

I am building a new proxy server with Squid & SquidClamAV.

I am facing problems with downloading large files. I have set a maxsize of 52428800 bytes in squidclamav.conf - which should prevent the scanning of files bigger than 50MB.

Downloading anything up to 2GB through the proxy works fine, but files bigger than that produce the following error: ICAP protocol error - The system returned: (32) Broken pipe

In c-icap server log i get this: Fri Dec 2 10:19:36 2011, general, Bug in the service. Please report to the servive author!!!!

Downloading a file at sizes below 2GB produces the download window instantly to the browser. As expected.

When starting to download anything above 2GB, it takes some time before producing the error. Following /var/tmp I see that it actually starts to download the file up to 50MB (the maxsize set in squidclamav.conf!) and then stops and produces the error.

I don't understand why this happens with files that are bigger than the maxsize in squidclamav.

The server is a CentOS 6 32bit machine. C-icap is compiled from source (v0.1.7), squid is from CentOS, clamd+squidclamav from repoforge.org. Below I've added related config files.

/etc/squid/squid.conf:

...
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_service service_req reqmod_precache \
     bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache \ 
     bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_resp allow all

/etc/squidclamav.conf:

maxsize 52428800
redirect http://localproxy/detecteio.php
clamd_local /var/run/clamav/clamd.sock
timeout 1
logredir 0
dnslookup 1

/etc/c-icap.conf:

PidFile /var/run/c-icap/c-icap.pid
CommandsSocket /var/run/c-icap/c-icap.ctl
Timeout 300
MaxKeepAliveRequests 100
KeepAliveTimeout 600
StartServers 3
MaxServers 10
MinSpareThreads     10
MaxSpareThreads     20
ThreadsPerChild     10
MaxRequestsPerChild  0
Port 1344
ServerAdmin root@localhost
ServerName localproxy.domain.local
TmpDir /var/tmp
MaxMemObject 131072
DebugLevel 1
ModulesDir /usr/local/lib/c_icap
ServicesDir /usr/local/lib/c_icap
TemplateDir /usr/local/share/c_icap/templates/
TemplateDefaultLanguage en
LoadMagicFile /usr/local/etc/c-icap.magic
RemoteProxyUsers off
RemoteProxyUserHeader X-Authenticated-User
RemoteProxyUserHeaderEncoded on
ServerLog /var/log/c-icap_server.log
AccessLog /var/log/c-icap_access.log
Service squidclamav squidclamav.so
Service echo srv_echo.so

/etc/clamd.conf:

LogFile /var/log/clamav/clamd.log
LogFileMaxSize 0
LogTime yes
LogSyslog yes
PidFile /var/run/clamav/clamd.pid
TemporaryDirectory /var/tmp
DatabaseDirectory /var/clamav
LocalSocket /var/run/clamav/clamd.sock
FixStaleSocket yes
TCPSocket 3310
TCPAddr 127.0.0.1
MaxConnectionQueueLength 30
MaxThreads 50
ReadTimeout 300
User clamav
AllowSupplementaryGroups yes
ScanPE yes
ScanELF yes
DetectBrokenExecutables yes
ScanOLE2 yes
ScanMail yes
ScanArchive yes
ArchiveBlockEncrypted no
3
  • This might not help at all, but the only time I've encountered an issue with downloading files bigger than 2GB was with 32bit internet explorer (which uses a single data stream)..
    – pauska
    Dec 2, 2011 at 12:09
  • What do your squid logs show for the failed downloads?
    – Flimzy
    Dec 3, 2011 at 18:30
  • Thanks for the replies. Squid log shows a TCP_MISS/500: TCP_MISS/500 3217 GET ftp.funet.fi/pub/Linux/INSTALL/Centos/6/isos/x86_64/… - DIRECT/193.166.3.2 text/html
    – mikkoko
    Dec 5, 2011 at 8:15

3 Answers 3

1

Generally, 32bit systems will have trouble dealing with files larger than 2GB, however, having the download capped at 50MB would suggest ClamAV is at fault.

Couple of suggestions;

1) Ensure Squid has been compiled with '--with-large-files'

2) Test the download locally on the box using curl or wget (rules out browser/network issue)

3) Try increasing the 'maxsize' directive in '/etc/squidclamav.conf' to see if you can download larger files.

4) Disable file scanning and re-test.

0

Based on the configuration provided, it seems that this issue is not because of squid as the configured limit is 50MB which is the limit to prevent scanning files larger than this but it has nothing to do with the size of download file. Can you please confirm the name and version of Web browser using for download, as some Internet explorer versions allow only less than specific size of files to download. For Example, this behavior may occur if you try to download a file that is larger than 2 GB in IE6 or is larger than 4GB in Internet Explorer 7 in this case you may switch to Firefox or any other web browser. Note: This download limit has been removed in IE8. Therefore, you should not experience this behavior in IE8. You can refer this link for more information http://support.microsoft.com/kb/298618 I hope this information will be helpful for you

2
  • Sorry but it's not a client/browser problem (bypassing the proxy allows me to download the tested files from the same client/browser). Somehow its squid, clamav or c-icap that is figuring the file size wrongly when the size exceeds 2GB.
    – mikkoko
    Dec 7, 2011 at 12:01
  • Have you tried by using Firefox, by doing some further investigation, I have found the same issue discussed in the following web link, please have a look: www2.us.squid-cache.org/mail-archive/squid-users/200705/… Dec 7, 2011 at 16:57
0

The proxy server was rebuilt with 64bit Centos 6. The same configuration was used, but the problem has gone away.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .