Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an Apache server behind an HAProxy server. Earlier today, the server became unavailable and HAProxy threw 503 errors on the front end. After investigating, it looks like iptables began blocking requests from HAProxy, and restarting Apache fixed the issue. I need help figuring out why iptables would randomly start denying these requests so I can prevent it from happening in the future.

HAProxy = 10.xxx.xxx.26
Web1 = 10.xxx.xxx.229

iptables log entry:

May 16 22:12:27 web1 kernel: [339449.200414] iptables denied: IN=eth1 
OUT= MAC=40:40:e9:0d:29:96:40:40:25:5e:3d:74:08:00 SRC=10.xxx.xxx.26 DST=10.xxx.xxx.229
LEN=80 TOS=0x00 PREC=0xC0 TTL=64 ID=9773 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.xxx.xxx.229 
DST=10.xxx.xxx.26 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=15095 DF PROTO=TCP SPT=80 DPT=43176 
WINDOW=55 RES=0x00 ACK FIN URGP=0 ]

iptables config (removed nagios/ssh rules for readability):

Web1:/var/log# cat /etc/iptables.test.rules 
*filter
:INPUT ACCEPT [5620:459239]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4375:4238642]
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -i ! lo -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

-A INPUT -i eth1 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -i lo -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i lo -p tcp -m tcp --dport 443 -j ACCEPT

-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j ACCEPT
COMMIT
share|improve this question

1 Answer 1

up vote 3 down vote accepted

This line is your culprit for the log entry:

-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

But the real juicy stuff is in the output. HAProxy is sending ICMP packets to your Web1 server, which is what you are seeing in the log.

The strange thing is the type... HAProxy is sending your web server ICMP Type 3/Code 3 packets, which is a response stating that a port is unreachable. This is a response from an earlier packet originating from your web server (port 80) to the HAProxy server (Dyn port 43176). What seems to have happened was that HAProxy ended a session and closed the dynamic port before Apache thought the conversation was over.

The reason why the above happened, I do not know. Take a look at timeouts maybe?

share|improve this answer
    
I did add a timeout http-request 4s to the HAProxy frontend based on a recommendation to prevent DDoS attacks (timeout client is set to 30s). I read that a low http-request timeout in HAProxy was important to have as it limits the amount of time the HEAD request can stay open. Is it possible that is the issue? –  Matt Beckman May 16 '11 at 23:59
    
That could definitely be it, try increasing it a bit. –  Hyppy May 17 '11 at 1:06
    
I removed the timeout http-request 4s over 24 hours ago, which would default that setting to the same setting as timeout client (30 seconds), and I haven't seen the issue since. Sweet! Do you think I am misunderstanding the intended use of timeout http-request or simply not using it correctly? haproxy.1wt.eu/download/1.4/doc/configuration.txt –  Matt Beckman May 18 '11 at 7:40
    
Honestly, I'm not exactly sure. Perhaps someone here a bit more knowledgeable could assist with this one? –  Hyppy May 18 '11 at 11:24
    
I guess I didn't give it enough time to surface again. It happened about an hour ago. I ended up whitelisting all traffic (including ICMP) on the LAN interface from the HAProxy server. It appeared to rear it's head only after a client successfully triggered a 504 Gateway Timeout. Hopefully my iptables update will permanently fix it, but it still feels like a cop-out. –  Matt Beckman May 18 '11 at 18:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.