Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When squid setup as TPROXY, returns ERR_CONNECT_FAIL on some sites . The server which squid runs on it is able to open those sites by lynx, wget, curl etc . Even if we manually set proxy in browser or use simple REDIRECT or DST-NAT, those sites could open.

From this page http://wiki.squid-cache.org/SquidFaq/InterceptionProxy

It causes path-MTU (PMTUD) to fail, possibly making some remote sites inaccessible. This is not usually a problem if your client machines are connected via Ethernet or DSL PPPoATM where the MTU of all links between the cache and client is 1500 or more. If your clients are connecting via DSL PPPoE then this is likely to be a problem as PPPoE links often have a reduced MTU (1472 is very common).

But i have same problem with ethernet

Here is tcpdump from one client:

Click here to see tcpdump when i use tproxy and the problem appears

and

Click here to see tcpdump when i manually set proxy in my browser and everyhting works fine

GET / HTTP/1.0
Host: 80.75.1.4
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Encoding: gzip, compress, bzip2
Accept-Language: en
User-Agent: Lynx/2.8.8dev.2 libwww-FM/2.14 SSL-MM/1.4.1

HTTP/1.0 504 Gateway Time-out
Server: squid
Mime-Version: 1.0
Date: Wed, 27 Feb 2013 15:39:03 GMT
Content-Type: text/html
Content-Length: 376353
X-Squid-Error: ERR_CONNECT_FAIL 110
X-Cache: MISS from cache.mysquid.com
X-Cache-Lookup: MISS from cache.mysquid.com:3128
Connection: close

ping -s 1500 80.75.1.4

PING 80.75.1.4 (80.75.1.4) 1500(1528) bytes of data.
1508 bytes from 80.75.1.4: icmp_req=1 ttl=58 time=5.28 ms
1508 bytes from 80.75.1.4: icmp_req=2 ttl=58 time=3.96 ms
1508 bytes from 80.75.1.4: icmp_req=3 ttl=58 time=4.28 ms

ping -s 1473 80.75.1.4 -M do

PING 80.75.1.4 (80.75.1.4) 1473(1501) bytes of data.
From 109.110.160.171 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 109.110.160.171 icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- 80.75.1.4 ping statistics ---
0 packets transmitted, 0 received, +2 errors

ping -s 1472 80.75.1.4 -M do

PING 80.75.1.4 (80.75.1.4) 1472(1500) bytes of data.
1480 bytes from 80.75.1.4: icmp_req=1 ttl=58 time=4.33 ms
1480 bytes from 80.75.1.4: icmp_req=2 ttl=58 time=4.32 ms

--- 80.75.1.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 4.320/4.329/4.338/0.009 ms

traceroute --mtu 80.75.1.4

traceroute to 80.75.1.4 (80.75.1.4), 30 hops max, 65000 byte packets
 1  x.x.x.10 (x.x.x.10)  0.820 ms F=1500  0.681 ms  0.243 ms
 2  y.y.y.1 (y.y.y.1)  2.969 ms  3.185 ms  2.994 ms
 3  217.218.181.193 (217.218.181.193)  2.836 ms  2.381 ms  2.487 ms
 4  217.218.185.22 (217.218.185.22)  3.617 ms  2.957 ms  3.176 ms
 5  78.38.119.237 (78.38.119.237)  2.050 ms  1.808 ms  2.264 ms
 6  217.11.30.250 (217.11.30.250)  3.522 ms  3.962 ms  2.674 ms
 7  * 80.75.1.4 (80.75.1.4)  3.507 ms *

tracepath 80.75.1.4

 1:  cache.mysquid.com                                     0.092ms pmtu 1500
 1:  x.x.x.10                                              0.380ms
 1:  x.x.x.10                                              0.309ms
 2:  y.y.y.1                                               3.390ms asymm  7
 3:  217.218.181.193                                       2.671ms asymm  5
 4:  217.218.185.22                                        2.944ms asymm  5
 5:  78.38.119.237                                         1.684ms
 6:  217.11.30.250                                         4.020ms
 7:  80.75.1.4                                             3.915ms reached
     Resume: pmtu 1500 hops 7 back 58

Also tried these steps http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses#Can.27t_connect_to_some_sites_through_Squid

I don't know related or not but i also have following configs

echo 1025 65000 > /proc/sys/net/ipv4/ip_local_port_range
echo 0 > /proc/sys/net/ipv4/tcp_syncookies
echo 131072 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/p2p1/rp_filter
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

and enabled/disabled following

#echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
#echo 0 > /proc/sys/net/ipv4/tcp_ecn

Here is TPROXY route rules

/sbin/iptables -t mangle -N DIVERT
/sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
/sbin/iptables -t mangle -A DIVERT -j ACCEPT
/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
/sbin/iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Note : Connection to internet providers is over GRE Tunnel with MTU 1476.

share|improve this question
    
I have had same Problem.Any body here to help? –  Abadis Feb 27 '13 at 14:45
2  
tcpdump it, why wouldn't ya? –  poige Feb 27 '13 at 15:02
1  
What you provided above is not tcpdump output. A tcpdump output (something like tcpdump -i $interface -vvv -n host 80.75.1.4) from when it works, and when it doesn't, should shed some light. –  abbe Mar 3 '13 at 9:00
    
@abbe You are right . link to tcpdump files added . –  Omid Kosari Mar 3 '13 at 10:04

3 Answers 3

From the tcpdump output you provided, following lines indicate a problem, i.e. your host sending a SYN to destination, and then sending an RST to the destination as well, but ofcourse TTLs are different, so seems like some TPROXY magic going on:

12:55:17.255717 IP (tos 0x0, ttl 64, id 11025, offset 0, flags [none], proto TCP (6), length 56)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [S], cksum 0x5f7e (incorrect -> 0x3c92), seq 572035649, win 14600, options [mss 1460,sackOK,TS val 66378265 ecr 0], length 0
12:55:17.258877 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [R], cksum 0xa779 (correct), seq 572035650, win 0, length 0
12:55:18.253670 IP (tos 0x0, ttl 64, id 11026, offset 0, flags [none], proto TCP (6), length 56)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [S], cksum 0x5f7e (incorrect -> 0x3b98), seq 572035649, win 14600, options [mss 1460,sackOK,TS val 66378515 ecr 0], length 0
12:55:18.257916 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [R], cksum 0xa779 (correct), seq 572035650, win 0, length 0

It doesn't seem like an MTU problem. To start with troubleshooting, I suggest removing TPROXY (assuming you're using that) and switch to REDIRECT based rule and see if problem goes away. It would be helpful if you could paste your REDIRECT or related iptables rules.

share|improve this answer
1  
switching to REDIRECT and the problem disappeared so it caused by TPROXY as you said . Now what we can do ? i have pasted iptable rules . –  Omid Kosari Mar 4 '13 at 8:22
    
I've never used TPROXY myself, but your routing rules look weird: ip route add local 0.0.0.0/0 dev lo table 100 Why is default route via loopback device ? –  abbe Mar 4 '13 at 21:59
    
Also I didn't notice before that you are using GRE tunnel. Could you provide routing table/rules (masking public IPs) to get a better idea of what your setup looks like ? And regarding tcpdump, could you provide them for the case when it's not working over the real interface instead of GRE interface, iff this host has the GRE interface. –  abbe Mar 5 '13 at 1:55
    
the "ip route add local 0.0.0.0/0 dev lo table 100" is from Squid Wiki and works for years . The GRE tunnel is not on squid server . it is on Router which is squid's gateway . squid connected to that router by gigabit ethernet. on that router i have a rule to CHANGE MSS to 1420 for many years . also tested to disable or decrease it . –  Omid Kosari Mar 5 '13 at 8:42
    
Is it possible for you to tcpdump on the router, basically to verify whether request goes out in case of TPROXY ? And also I assume this TPROXY box is in front-of users, so all responses coming from remote endpoint with users' IP (like FTP data request) gets through this TPROXY box, so a network setup something like [internet] --- [gre-router] --- [tproxy box] --- [user1,user2,user3,...userN]. Right? –  abbe Mar 7 '13 at 2:30

The Unix error 110 means "Connection timed out". The remote site never responded to the request.

When I tried to reach 80.75.1.4, I also was not able to reach it. It appears from ping and traceroute that there is significant packet loss on the network after it enters the state of Iran. This is probably why the site is failing to respond.

$ traceroute 80.75.1.4
traceroute to 80.75.1.4 (80.75.1.4), 30 hops max, 60 byte packets
 1  hosted.by.leaseweb.com (198.7.57.254)  1.275 ms  1.227 ms  1.888 ms
 2  108.59.15.23 (108.59.15.23)  0.249 ms  0.243 ms  0.216 ms
 3  be4.cr2.wdc1.leaseweb.net (108.59.15.108)  0.922 ms be3.cr2.wdc1.leaseweb.net (108.59.15.102)  0.922 ms ae1.cr1.wdc1.leaseweb.net (108.59.15.116)  0.275 ms
 4  xe-7-2-0-690.edge1.Washington1.Level3.net (4.28.127.57)  1.258 ms xe-5-3-0-673.edge3.Washington1.Level3.net (4.59.144.161)  1.181 ms  1.247 ms
 5  ae-4-90.edge1.Washington4.Level3.net (4.69.149.207)  1.779 ms  1.749 ms ae-3-80.edge1.Washington4.Level3.net (4.69.149.143)  1.670 ms
 6  ash-bb1-link.telia.net (213.248.81.117)  1.499 ms ash-bb1-link.telia.net (213.248.89.177)  15.220 ms ash-bb1-link.telia.net (213.248.103.233)  1.505 ms
 7  ash-bb4-link.telia.net (213.155.130.58)  1.560 ms  1.637 ms *
 8  ldn-bb2-link.telia.net (80.91.246.67)  76.651 ms  76.782 ms  76.761 ms
 9  ldn-b1-link.telia.net (80.91.248.93)  77.581 ms  77.552 ms  77.515 ms
10  telecominfra-ic-132493-ldn-b1.c.telia.net (213.248.76.42)  234.204 ms  233.666 ms  233.731 ms
11  * * *
12  * 10.10.53.222 (10.10.53.222)  251.938 ms *
13  217.11.30.246 (217.11.30.246)  250.353 ms  250.725 ms  250.322 ms
14  80.75.1.4 (80.75.1.4)  240.361 ms  240.269 ms  240.327 ms

What's worse, is it looks like someone has used RFC 1918 addresses on the public Internet in this path. Among other things, this causes Path MTU discovery to break, which means many TCP connections will also break.

To quote from one source, Cisco's IP Addressing Best Practices (PDF):

Any router-to-router links connecting to areas of the network with public addressing should be addressed with public IP addresses. Routers serving specific areas of the network using and continuing to use only private addresses may use private addresses on the router-to-router links.

This requirement enables and helps ensure that path MTU discovery (RFC 1191) works properly; routers must be able to send "packet-too-big" errors and must be assured that the packets are likely to arrive at the original source host. If router-to-router links are addressed with RFC 1918 addresses, the Internet Control Message Protocol (ICMP) messages generated by the router will come from an RFC 1918 address. Networks filtering out incoming packets with RFC 1918 source IP addresses, or using unicast reverse path forwarding (uRPF), will likely drop these packets, breaking TCP for those applications. This will cause large packet transfer across a TCP connection to fail completely or perform suboptimally.


In short, there are multiple network issues here which will cause the site to be unreachable for many or most Internet users.

share|improve this answer
1  
Thanks but unfortunately the private address in not the case here . Iran gov uses private ip as huge intranet in country . i have thousands of such sites without problem because the squid server is also in that intranet . –  Omid Kosari Feb 28 '13 at 6:53
    
They really shouldn't do that... but it's clear that the network is the cause of the issue and there's not much you can do from either endpoint. –  Michael Hampton Feb 28 '13 at 6:54
    
So why the squid server itself is able to open that site ? also why everything works if i manually set that proxy in my browser ? also there are many websites outsite of Iran with same problem . –  Omid Kosari Feb 28 '13 at 7:19

The prossible problems could be MTU and also routing problems, as you could open addresses via links or curl the issue may related to MTU.

To check MTU you need to use ping to check correct Path MTU and manually set correct MTU to your network adapter. also you can use Change-MSS in your iptables to fix this issue I try yo explain both solutions:

for MTU checking try to ping destination with the following command:

ping -M do -s 1472 <ip address>

the -s argeument sets ping package size you may alter this value (normally decrease) until you find the value at that ping your packages start to be fragmented. then add 28 bytes (ping header size) to this value and this is your PMTU size. you may also use tracepath to discover the PMTU.

then you must set a iptables rule to set change packets MSS size accourding to your Path MTU size.

 iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss <PMTU size>

you can also use --clamp-mss-to-pmtu to ask iptables to check for PMTU automatically:

 iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Thats all I can say right now, hope it helps

P.S. Do you use TProxy for transparent proxy?

share|improve this answer
1  
I have already tried what you said with no success . Yes i am using tproxy . –  Omid Kosari Mar 3 '13 at 15:51
1  
@Boynux : Did you just Copy and Paste the question in to answer? He has just mentioned this.Please read questions and then try to answer –  Abadis Mar 4 '13 at 13:13
    
@Abadis first. the question edited after my post and second. even in edited question nothing mentioned about MSS values. So Please read question and answers carefully before posting comments. –  Boynux Mar 5 '13 at 13:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.