iptables connlimit triggering unexpectedly

Question

I've been using these firewall rules:

-I FORWARD -p tcp --syn -m connlimit --connlimit-above 50 -j REJECT
-I FORWARD -p tcp --syn -m connlimit --connlimit-above 50 -j LOG --log-prefix "CONNLIMIT: " --log-level debug

it seems straightforward enough: prevent someone from opening more than 50 connections and causing a denial of service. I've tested it successfully against slowloris. I turned the limit up to 50 specifically to prevent issues with false-positives (Apache can be very connection-hungry.) however, this morning, I get an email from my Nagios monitor and my logs show several lines of "CONNLIMIT" with the source IP being my monitoring system.

I have no idea why this is happening. at most, my monitoring server should be performing 5-10 checks and perhaps a ping or SSH connection. I'd be shocked if I had more than 25 conncetions open, yet 2 weekends in a row, I've managed to trigger connlimit 50 and rudely awaken myself.

is there something wrong with my firewall rules? (maybe add the 'new' flag?) is Nagios not closing its connections properly? I'm not even sure how to continue debugging this issue without logging every packet on the wire and patiently waiting for my cell phone to go off at some awful hour.

[edit: just for fun, here's the server logs]

Oct  9 11:33:22 adapt kernel: [1888526.442640] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=2076 DF PROTO=TCP SPT=46536 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Oct  9 11:34:22 adapt kernel: [1888586.443869] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=43048 DF PROTO=TCP SPT=57931 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Oct  9 11:35:42 adapt kernel: [1888667.011376] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=19161 DF PROTO=TCP SPT=63669 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
Oct  9 11:35:48 adapt kernel: [1888673.093663] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=48302 DF PROTO=TCP SPT=63673 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
Oct  9 11:35:53 adapt kernel: [1888678.361267] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=11711 DF PROTO=TCP SPT=63677 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
Oct  9 11:36:04 adapt kernel: [1888688.517868] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=6316 DF PROTO=TCP SPT=44206 DPT=443 WINDOW=5840 RES=0x00 SYN URGP=0
Oct  9 11:36:21 adapt kernel: [1888705.382273] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=29613 DF PROTO=TCP SPT=63697 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
Oct  9 11:36:49 adapt kernel: [1888733.467511] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=52433 DF PROTO=TCP SPT=40930 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
Oct  9 11:37:04 adapt kernel: [1888748.574700] CONNLIMIT: IN=eth0 OUT=eth1 SRC=[MONITOR] DST=[HOST] LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=26329 DF PROTO=TCP SPT=44223 DPT=443 WINDOW=5840 RES=0x00 SYN URGP=0

we can see that it's checking a few ports and issuing a check about once a minute.

Answer 1

You probably want recent not connlimit.

Here's an example from one of my hosts that limits SSH connections:

iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -j SSH_Brute_Force
iptables -A SSH_Brute_Force -m recent --set --name SSH --rsource
iptables -A SSH_Brute_Force -m recent ! --update --seconds 120 --hitcount 5 --name SSH --rsource -j RETURN
iptables -A SSH_Brute_Force -m recent --update --name SSH --rsource
iptables -A SSH_Brute_Force -p tcp -j DROP

Or:

• Only SSH packets have already been selected for entry into the chain
• Assign connections to the "SSH" recent queue
• Set a 120 second timeout, and a maximum of five new connections in that time, return the calling chain sessions that shouldn't be limited (calling chain then allows to good hosts)
• Update the connection list
• Drop packet

Answer 2

When the rule is triggered, do you see open connections from the monitor host in netstat?

What about in /proc/net/ip_conntrack on the router?

ISTR that connlimit actually limits all connections that hit this rule, and this is a very general rule. Is the number going up towards 50 due to other connections going through the router?

How about adding a debug rule to determine what's actually happening:

-I FORWARD -p tcp --syn -s [MONITOR] -j LOG --log-prefix "MONITOR CONNECT: " --log-level debug