5

I run a webserver (Apache) on a Debian 7 machine, with iptables on the same machine. iptables rules are generated by the ConfigServer Firewall (CSF) script. The websites hosted there are fine on my end, however I'm seeing a lot of dropped inbound traffic on port 80.

Here's an extract from the logs (webserver IP: 11.22.33.44):

Jan 27 15:21:36 [hostname] kernel: [1229124.817624] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=199.30.24.209 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=3144 DF PROTO=TCP SPT=36879 DPT=80 WINDOW=510 RES=0x00 ACK FIN URGP=0
Jan 27 15:21:36 [hostname] kernel: [1229124.872795] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=199.30.24.209 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=3183 DF PROTO=TCP SPT=36684 DPT=80 WINDOW=513 RES=0x00 ACK FIN URGP=0
Jan 27 16:03:36 [hostname] kernel: [1231642.223513] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=5.39.50.0 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=122 ID=19101 DF PROTO=TCP SPT=51394 DPT=80 WINDOW=508 RES=0x00 ACK FIN URGP=0
Jan 27 16:03:41 [hostname] kernel: [1231647.015463] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=5.39.50.0 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=122 ID=26215 DF PROTO=TCP SPT=51394 DPT=80 WINDOW=508 RES=0x00 ACK RST URGP=0
Jan 27 16:03:51 [hostname] kernel: [1231656.677627] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=5.39.50.0 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=122 ID=10630 DF PROTO=TCP SPT=51394 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Jan 27 16:04:42 [hostname] kernel: [1231707.911962] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=86.217.34.8 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=111 ID=3513 DF PROTO=TCP SPT=54465 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
Jan 27 16:04:42 [hostname] kernel: [1231707.911976] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=86.217.34.8 DST=11.22.33.44 LEN=40 TOS=0x00 PREC=0x00 TTL=111 ID=3514 DF PROTO=TCP SPT=54465 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

Hundreds of line of that stuff. No specific timing pattern, and the drop occurs for random clients.

But these dropped packets are always flagged ACK FIN, RST, ACK RST, and much less often SYN. I understand that that these are "acknowledge transfer and end connection" packets.

Relevant iptables rules:

Chain INPUT (policy DROP)
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:80
LOGDROPIN  all  --  anywhere             anywhere

Chain LOGDROPIN (1 references)
LOG        tcp  --  anywhere             anywhere             limit: avg 30/min burst 5 LOG level warning prefix "Firewall: *TCP_IN Blocked* "
DROP       all  --  anywhere             anywhere

So by the looks of it, these packets are being dropped because their corresponding connection is no longer open, i.e. RELATED or ESTABLISHED. Which means that the transfer succeeds, and that the server is closing the connection before the client has time to acknowledge received data, thus leaving the connection (and maybe the HTTP request?) hanging on the client side.

I'm really curious as to what could be causing this. And also wondering if clients are being affected, as I can't seem to replicate the issue. I'm hoping that you guys can help me understand!

Thanks for reading =)

EDIT: OK I went fishing and made a packet trace for one of these cases where [ACK,FIN] packets get dropped (following the method described in my comment below). The three last packets are the ones that got dropped by the firewall:

Source                Destination           Protocol Length Info                                                                            Time
[CLIENT COMP]         [WEBSERVER]           TCP      68     55478 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1400 WS=4 SACK_PERM=1               2014-01-29 20:44:36.044009
[WEBSERVER]           [CLIENT COMP]         TCP      68     http > 55478 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=512 2014-01-29 20:44:36.044052
[WEBSERVER]           [CLIENT COMP]         TCP      68     http > 55478 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=512 2014-01-29 20:44:37.243948
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [ACK] Seq=1 Ack=1 Win=16800 Len=0                                  2014-01-29 20:44:37.421123
[CLIENT COMP]         [WEBSERVER]           HTTP     464    GET /sites/default/files/js/js_R9UbiVw2xuTUI0GZoaqMDOdX0lrZt....js HTTP/1.1     2014-01-29 20:44:37.432097
[WEBSERVER]           [CLIENT COMP]         TCP      56     http > 55478 [ACK] Seq=1 Ack=409 Win=15872 Len=0                                2014-01-29 20:44:37.432122
[WEBSERVER]           [CLIENT COMP]         HTTP     963    HTTP/1.1 200 OK  (text/javascript)                                              2014-01-29 20:44:37.432703
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [ACK] Seq=409 Ack=908 Win=15892 Len=0                              2014-01-29 20:44:37.769928
[WEBSERVER]           [CLIENT COMP]         TCP      56     http > 55478 [FIN, ACK] Seq=908 Ack=409 Win=15872 Len=0                         2014-01-29 20:44:42.437129
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [ACK] Seq=409 Ack=909 Win=15892 Len=0                              2014-01-29 20:44:42.580378
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [FIN, ACK] Seq=409 Ack=909 Win=15892 Len=0                         2014-01-29 20:49:44.668730
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [FIN, ACK] Seq=409 Ack=909 Win=15892 Len=0                         2014-01-29 20:49:49.194316
[CLIENT COMP]         [WEBSERVER]           TCP      56     55478 > http [FIN, ACK] Seq=409 Ack=909 Win=15892 Len=0                         2014-01-29 20:49:58.869659

According to this:

  1. The server sends its HTTP 200 OK response
  2. Wishing to close the connection. It sends out an [FIN, ACK] packet to the client, as I'm guessing it's trying to make a simultaneous connection close.
  3. The client responds with an [ACK] to the server's [FIN, ACK] (The Seq/Ack numbers are in the correct order) but no [FIN], which according to the spec means that the connection is only half-closed.
  4. But then, 5 minutes later, the client sends a [FIN, ACK], which is unusual because it already sent an ACK to the server's "end of connection'. I'm guessing some form of TCP timeout is involved, and that the client is only trying to close a connection that was left open.

So clearly the connection termination handshake isn't occuring in the correct sequence. And it looks like the problem is on the client side, who doesn't close the connection until the 5 minute timeout.

I noticed 3 things:

  • This particular client was running a modern browser, so the browser's not the issue.
  • He/she was sending/receiving a lot of TCP retransmissions, suggesting a lossy connection. It turns out, that this guy is connected from Morocco in 3G, so it does add up ^^.
  • Finally, the Apache logs show that he did get the correct HTTP response, so site traffic is not affected by the "issue".

Interesting stuff. If you have anything, shoot!

4
  • May be that you have some odd behavior with half-closed connections being seen as no longer established?
    – NickW
    Jan 27, 2014 at 16:21
  • I was thinking it was something like that, yeah. I guess I should look into how the connection takes place. Actually looking for a way to log the full TCP connection for any address whose ACK FIN/RST traffic gets dropped. Unfortunately I have no idea how to do that! Jan 28, 2014 at 14:06
  • For future reference: There's actually an efficient way to do just that. Run tcpdump -w /path/to/dump.pcap tcp port 80 on the server until one of these random ACK FIN drops occur (monitor the logs to see that event). Stop the tcpdump, pull that pcap file and open it in Wireshark. Filter by the ip address(es) for which the drop occured. Look at the connection opening and closing handshakes, and compare them to a "healthy" connection that you logged yourself by opening your website. Will add answers as soon as I'm done troubleshooting! Jan 29, 2014 at 11:42
  • That's a little bit interactive, but still it should do what you need :)
    – NickW
    Jan 29, 2014 at 11:52

1 Answer 1

1

According to a mail chain on netfilter, it appears to be a normal function of iptables. If you want to not get those messages in the log, you could try adding INVALID to the list of states to accept on port 80 and see if that problem goes away.

Otherwise, other than being annoying and log-filling, it's harmless traffic and your clients wouldn't be experiencing any problems.

1
  • Thanks for the answer =) It's most probably harmless, yes, as the data is still being transmitted correctly. The guy on that mailchan, however, was talking about outgoing ACK PSH FIN dropped packets, which means the client closes the connection first. I'm seeing the same, but inbound, which would mean the server is ending the connection prematurely and leaving the client hanging in a half open TIME_WAIT/CLOSE_WAIT state. But as the mail chan states, it's most probably a connection timeout that forces the connection out of the state table before the client has time to acknowledge closing it. Jan 28, 2014 at 14:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .