Why did iptables suddenly block HAProxy

Question

I have an Apache server behind an HAProxy server. Earlier today, the server became unavailable and HAProxy threw 503 errors on the front end. After investigating, it looks like iptables began blocking requests from HAProxy, and restarting Apache fixed the issue. I need help figuring out why iptables would randomly start denying these requests so I can prevent it from happening in the future.

HAProxy = 10.xxx.xxx.26
Web1 = 10.xxx.xxx.229

iptables log entry:

May 16 22:12:27 web1 kernel: [339449.200414] iptables denied: IN=eth1 
OUT= MAC=40:40:e9:0d:29:96:40:40:25:5e:3d:74:08:00 SRC=10.xxx.xxx.26 DST=10.xxx.xxx.229
LEN=80 TOS=0x00 PREC=0xC0 TTL=64 ID=9773 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.xxx.xxx.229 
DST=10.xxx.xxx.26 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=15095 DF PROTO=TCP SPT=80 DPT=43176 
WINDOW=55 RES=0x00 ACK FIN URGP=0 ]

iptables config (removed nagios/ssh rules for readability):

Web1:/var/log# cat /etc/iptables.test.rules 
*filter
:INPUT ACCEPT [5620:459239]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4375:4238642]
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -i ! lo -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

-A INPUT -i eth1 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -i lo -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i lo -p tcp -m tcp --dport 443 -j ACCEPT

-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j ACCEPT
COMMIT

Answer 1

This line is your culprit for the log entry:

-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

But the real juicy stuff is in the output. HAProxy is sending ICMP packets to your Web1 server, which is what you are seeing in the log.

The strange thing is the type... HAProxy is sending your web server ICMP Type 3/Code 3 packets, which is a response stating that a port is unreachable. This is a response from an earlier packet originating from your web server (port 80) to the HAProxy server (Dyn port 43176). What seems to have happened was that HAProxy ended a session and closed the dynamic port before Apache thought the conversation was over.

The reason why the above happened, I do not know. Take a look at timeouts maybe?