Why firewall/server becomes slow after 1-2 days? Affects firewall and machines behind it

Question

After 1 or 2 days, my firewall becomes extremely slow. I have 5 machines behind it and they also become extremely slow. After hardware reboot, it becomes fast again.

My current conntrack limit:

[root@fw ~]# cat /proc/sys/net/ipv4/ip_conntrack_max
100000

Average conntrack table count:

[root@fw ~]# cat /proc/net/ip_conntrack |wc -l
1301

My iptables rules:

    [root@fw ~]# iptables -L -n
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination

    Chain FORWARD (policy DROP)
    target     prot opt source               destination
    DROP       all  --  24.0.0.0/8           0.0.0.0/0
    ACCEPT     udp  --  $IP_155        0.0.0.0/0           udp
    ACCEPT     udp  --  0.0.0.0/0            $IP_155       udp dpt:8888
    ACCEPT     udp  --  0.0.0.0/0            $IP_155       udp dpt:9999
    ACCEPT     udp  --  0.0.0.0/0            $IP_155       udp dpt:8777
    ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0
    ACCEPT     all  --  $IP_160        $IP_39
    ACCEPT     all  --  $IP_39         $IP_160
    ACCEPT     all  --  $ip_veloX        0.0.0.0/0
    ACCEPT     all  --  192.168.1.0/24       0.0.0.0/0
    ACCEPT     all  --  192.168.1.0/24       0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            192.168.1.0/24
    ACCEPT     all  --  192.168.1.0/24       0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            192.168.1.0/24
    ACCEPT     all  --  192.168.1.0/24       0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            192.168.1.0/24
    ACCEPT     tcp  --  0.0.0.0/0            $IP_39        state NEW tcp dpt:2106
    ACCEPT     tcp  --  0.0.0.0/0            $IP_160       state NEW tcp dpt:2106
    ACCEPT     tcp  --  0.0.0.0/0            $IP_156       state NEW multiport dports 7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_211       state NEW multiport dports 7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_215       state NEW multiport dports 7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_160       state NEW multiport dports 7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_155       state NEW multiport dports 7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_155       state NEW multiport dports 8767,9999,8777,14534,51234,6969,7777
    ACCEPT     tcp  --  0.0.0.0/0            $IP_160       state RELATED,ESTABLISHED
    ACCEPT     tcp  --  0.0.0.0/0            $IP_156       state RELATED,ESTABLISHED
    ACCEPT     tcp  --  0.0.0.0/0            $IP_211       state RELATED,ESTABLISHED
    ACCEPT     tcp  --  0.0.0.0/0            $IP_215       state RELATED,ESTABLISHED
    ACCEPT     tcp  --  0.0.0.0/0            $IP_39        state RELATED,ESTABLISHED
    ACCEPT     tcp  --  0.0.0.0/0            $IP_155       state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_160        0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_156        0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_211        0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_215        0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_39         0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_155        0.0.0.0/0           state RELATED,ESTABLISHED
    ACCEPT     tcp  --  $IP_160        0.0.0.0/0           state NEW
    ACCEPT     tcp  --  $IP_156        0.0.0.0/0           state NEW
    ACCEPT     tcp  --  $IP_211        0.0.0.0/0           state NEW
    ACCEPT     tcp  --  $IP_215        0.0.0.0/0           state NEW
    ACCEPT     tcp  --  $IP_39         0.0.0.0/0           state NEW
    ACCEPT     tcp  --  $IP_155        0.0.0.0/0           state NEW

Chain OUTPUT (policy ACCEPT)

    target     prot opt source               destination
    [root@fw ~]#

[root@fw ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 142G 21G 114G 16% /

Also what is your free disk space? You aren't filling up because of some log file, are you? — Andy May, Jun 5, 2009 at 12:34
I have enough disk space.. [root@fw ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 142G 21G 114G 16% / tmpfs 501M 0 501M 0% /dev/shm I have 1GB RAM.. I believe it can indeed be a RAM issue.. I will check it when lag arrives.. Thanks!! — Filipe, Jun 5, 2009 at 12:41
and yes - what exactly slow menas? irresponsive when used from console? high latency on local lan segment? — pQd, Jun 5, 2009 at 13:46

pQd · Accepted Answer · 2009-06-05 12:45:25Z

3

it'll not help directly in this case but always put

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

as first rule. this will save you some cpu cycles for all packets belonging to connections already established.

similarly with input / output if you have DROP policy there.

what is load of the machine [ take it from w or top ]

do you have something interesting in dmesg?

do you do some address translation?

answered Jun 5, 2009 at 12:45

pQd

30.2k6 gold badges66 silver badges110 bronze badges

Thanks for the tip! Load is always under ~ 0.3 dmesg seems to be ok.. The only other rule i forgot is this: [root@fw ~]# iptables -t mangle -L Chain PREROUTING (policy ACCEPT) target prot opt source destination TOS tcp -- anywhere anywhere tcp dpt:cbt TOS set Minimize-Delay I think it is up to RAM.. I am increasing it and hopefuly it will not happen again.. thank you all for the help.. first time i use this site and all i can say is this site is amazing
– Filipe
Jun 5, 2009 at 13:06
1

@Filipe : install munin munin.projects.linpro.no it's amazing how many things become apparent when you take a look at 'big picture' with weekly/monthly trends.
– pQd
Jun 5, 2009 at 13:24

Add a comment |

buecking · Accepted Answer · 2009-06-05 13:06:40Z

2

Provided that you're referring to your bandwidth becoming slow, I would first try reloading iptables (root$ /etc/init.d/iptables reload, or alternatively flushing and reloading your rules manually) and see if that solves your problem. If it does, then you know it's your fw config. If it doesn't then you can start monitoring your system to see if something is eating up resources.

I'm guessing the former problem, and it's generally related to your nat translation table getting full. You can view it with '-t nat' and whatever appropriate options.

answered Jun 5, 2009 at 13:06

buecking

7031 gold badge6 silver badges10 bronze badges

+1 for solid troubleshooting! Check individual services to isolate the problem. Are there other services he might try? And if iptables is the culprit is there a setting change to limit the log size or periodically flush it?
– tomjedrz
Jun 5, 2009 at 13:43

Add a comment |

David · Accepted Answer · 2009-06-05 15:24:16Z

2

Along with the other suggested possibilities, another one is that one of your gateways NICs is starting to fail. When things start to "slow down", run and ifconfig and check the packet counts. In particular check the error counts.

answered Jun 5, 2009 at 15:24

David

3,59523 silver badges17 bronze badges

or try re-initializing the interfaces (down them, then up them)
– derobert
Jun 5, 2009 at 18:08

Add a comment |

Stack Exchange Network

Why firewall/server becomes slow after 1-2 days? Affects firewall and machines behind it

3 Answers 3

You must log in to answer this question.

Hot Network Questions

Why firewall/server becomes slow after 1-2 days? Affects firewall and machines behind it

3 Answers 3

You must log in to answer this question.

Related

Hot Network Questions