Trying to make iptables stateless is causing unforeseen filtering

Question

I'm trying to improve the performance of my server by tuning iptables to not keep track of TCP connection's state. I'm looking at this guide: http://cotdp.com/2011/07/nginix-on-a-256mb-vm-slice-24000-tps/

However, if I do any of the following it seems like all outgoing connections are cut off:

remove this rule: INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT adding these:

iptables -t raw -I OUTPUT -j NOTRACK
iptables -t raw -I PREROUTING -j NOTRACK

Immediately after making either change makes it so "ping google.com" returns an error about not being able to find "google.com" (ie the DNS stops resolving).

Here are the rules that are loaded up on boot, but other rules are then added by fail2ban:

*filter
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp --dport ssh -j ACCEPT
-A INPUT -p tcp --dport http -j ACCEPT
-A INPUT -p tcp --dport https -j ACCEPT
-A INPUT -p tcp --dport smtp -j ACCEPT
-A INPUT -p tcp --dport ssmtp -j ACCEPT
-A INPUT -j REJECT
-A FORWARD -j REJECT
COMMIT

Here's the output of iptables --list:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-ssh  tcp  --  anywhere             anywhere            multiport dports ssh 
fail2ban-ssh-ddos  tcp  --  anywhere             anywhere            multiport dports ssh 
fail2ban-pam-generic  tcp  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             loopback/8          reject-with icmp-port-unreachable 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssmtp 
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain fail2ban-pam-generic (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain fail2ban-ssh-ddos (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Answer 1

You have a rule which blocks all incoming traffic:

-A INPUT -j REJECT

And you stop connection tracking, so the rule to accept established connections' packets doesnt work anymore:

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

So your DNS packet goes out, is not tracked, and is then rejected by the first rule.

You need to enable tracking for the second rule to work, or add rules to allow incoming traffic from "good" sources.

Answer 2

Linux connection tracking doesn't just track TCP connections, it also tracks UDP psuedo-connections and various other things.

Without the notrack rules what happens is.

• Your system sends a DNS request, this creates a connection tracking entry.
• The DNS server generates a reply
• The reply matches your "accept established and related" rule.

With the notrack rule what happens is

• Your system sends a DNS request, but does not create a connection tracking entry.
• The DNS server generates a reply
• The reply does not match your "accept established and related" rule.
• The reply falls through and hits your catch-all reject rule.

So how to fix this? Either you have to explicitly allow response traffic without relying on your state tracking or you have to get more selective about your notrack rules.

For DNS explicitly allowing response traffic is probablly reasonable. You know what your DNS server is and you probablly trust it.

If your server needs to access resources on the internet in general then going compeletely without connection tracking means going with a far less effective firewall.

I expect you could get much of the performance benefit with far less pain by only applying notrack to traffic to and from your webserver. e.g.

iptables -t raw -I OUTPUT -p tcp --sport http -j NOTRACK
iptables -t raw -I PREROUTING -p tcp --dport http -j NOTRACK