iptables drops some packets on port 80 and i don't know the cause

Question

We are running a firewall with iptables on our Debian Lenny system. I show you only the relevant entries of our firewall.

Chain INPUT (policy DROP 0 packets, 0 bytes)
target  prot opt in out  source     destination         
ACCEPT  all  --  lo *    0.0.0.0/0  0.0.0.0/0
ACCEPT  all  --  *  *    0.0.0.0/0  0.0.0.0/0  state RELATED,ESTABLISHED
ACCEPT  tcp  --  *  *    0.0.0.0/0  0.0.0.0/0  tcp dpt:80 state NEW

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
target  prot opt in out  source     destination
ACCEPT  all  --  *  lo   0.0.0.0/0  0.0.0.0/0           
ACCEPT  all  --  *  *    0.0.0.0/0  0.0.0.0/0  state RELATED,ESTABLISHED 
LOGDROP all  --  *  *    0.0.0.0/0  0.0.0.0/0

Some packets get dropped each day with log messages like this:

Feb 5 15:11:02 host1 kernel: [104332.409003] dropped IN= OUT=eth0 SRC=<OWN_IP> DST=<REMOTE_IP> LEN=1420 TOS=0x00 PREC=0x00 TTL=64 ID=18576 DF PROTO=TCP SPT=80 DPT=59327 WINDOW=54 RES=0x00 ACK URGP=0

for privacy reasons I replaced IP Addresses with <OWN_IP> and <REMOTE_IP>

This is no reason for any concern, but I just want to understand what's happening. The web server tries to send a packet to the client, but the firewall somehow came to the conclusion that this packet is "UNRELATED" to any prior traffic.

I have set a kernel parameter ip_conntrack_ma to a high enough value to be sure to get all connections tracked by iptables state module:

sysctl -w net.ipv4.netfilter.ip_conntrack_max=524288

What's funny about that is I get one connection drop every 20 minutes:

06:34:54
06:52:10 
07:10:48 
07:30:55 
07:51:29 
08:10:47 
08:31:00 
08:50:52
09:10:50
09:30:52 
09:50:49 
10:11:00 
10:30:50 
10:50:56 
11:10:53 
11:31:00 
11:50:49 
12:10:49 
12:30:50 
12:50:51 
13:10:49 
13:30:57 
13:51:01 
14:11:12
14:31:32 
14:50:59 
15:11:02 

That's from today, but on other days it looks like this, too (sometimes the rate varies).

What might be the reason?

Any help is greatly appreciated. kind regards Janning

Answer 1

I have seen a very similar issue on my own systems and I may have an answer as to what is happening.

Apparently, TCP allows the client to send a FIN packet, and you to ACK that FIN packet, but still keep your end of the connection open and push more data through it, sending your own FIN packet at some time in the future. I remember reading that this was implemented at some point in Kernel 2.6 although my memory on this point may be inaccurate/irrelevant.

Many firewalls, including IPTables, don't seem to have implemented this yet or implement it incorrectly and consider the connection closed as soon as they see a FIN followed by an ACK. The result of this is that every so often, my server sends a packet out to a client that the firewall thinks is not part of an established connection and is not new and so drops it.

In practice, I haven't seen any important data in those final few packets that are being dropped, so the effect is a few extra reset packets getting thrown around and some extra connections left in the FIN_WAIT state but no broken or half-loaded pages.

I don't know about the 20 minute timings you are seeing but I would guess that it's a client of some sort that runs regularly every 20 minutes and does a request that causes your server to do the FIN ACK PSH FIN RST sequence.

This post may offer more information about exactly why IPTables drops these packets: http://lists.netfilter.org/pipermail/netfilter/2005-August/062059.html

According to Chris Brenton, it's a mismatch between the timeouts of the client, the server and the firewall for packets in the one-side-closed state.

Answer 2

Never tried this myself, but maybe using these instructions to log the entire packet in pcap might help you find your answer.

Answer 3

Does the originating/source IP show up in that log output? If yes does that IP show any valid requests in the http logs? Perhaps a monitoring system of some kind is checking http on your server, since you said it was in consistent intervals. Just throwing stuff out there.

Answer 4

What I do not understand is: how come the iptables log messages show packets that are dropped while outgoing, while the latter log shows dropped packets ingoing?

I suspect you might have two separate issues here. Perhaps somebody is trying to spoof your address, thus an unrelated reply to (the spoofed address) is caught by iptables and dropped. This would cover the outgoing drops.

Regarding the incoming drops, the 20mins time interval is suspicious.. perhaps ARP? Some DHCP authentication..? 20 mins is 1200 seconds; perhaps there are 3 separate chains going on, each adding up to 3600 (default DHCP lease time), but you only see the single packets. Just a hint.

Answer 5

If someone tries to scan you by sending ACK packets and your firewall requires a STATE(an established connection) it will get dropped.

Stateless firewalls only drops incoming SYN packets and lets ACK through. This means you can scan behind the firewall even though the port is blocked. How? Since the ACK is not recognized the system will play nice and send a RST(RESET) packet telling you that we dont have a connection. You now know that something is listening on that port.

And looking at the information you provided it is indeed an ACK packet thats being dropped.

You can confirm this by using nmap(from an outside system):
nmap -sA -p80 your_ip