My server just had a 2 minute downtime. It was serving around 400requests/second. The only odd thing I found in the munin reports was: http://monitor.wingify.com/munin/visualwebsiteoptimizer.com/lb1.visualwebsiteoptimizer.com-fw_forwarded_local.html

How should I interpret it and what exactly was the issue and how to make sure this doesnt happen again.

There is nothing unusual in the logs. What and how should I debug.


In your syslog do you see:

ip_conntrack: table full, dropping packet.

If so, your conntrack table is full. You can also check the current number of connections tracked using this command:

username@localmachine:~$ cat /proc/slabinfo|grep conntrack
nf_conntrack_expect      0      0    248   16    1 : tunables    0    0    0 :   slabdata      0      0      0
nf_conntrack_ffffffff81d11da0   1499   1536    320   12    1 : tunables    0    0    0 : slabdata    128    128      0

The first/second number is the value that the system is currently tracking (active/total). You can check the current max number of connections your kernel can watch using this command:

sysctl net.netfilter.nf_conntrack_max

For CentOS/Redhat, it is called ip_conntrack. If you don't see nf_conntrack_max check this:

sysctl net.ipv4.netfilter.ip_conntrack_max

You can increase using the sysctl -w command:

sysctl -w net.netfilter.nf_conntrack_max=131072

For example. The number is dependent on how much memory you have in the machine and whether the kernel can store the connection informations. The above setting doubles the default Ubuntu setting of 65536.

When you find a good value, add it to your /etc/sysctl.conf file to apply it on reboots:

# increase the number of conntrack max

As long as your machine and the upstream network equipment can handle the traffic, you should be good. As you had mentioned it is good to see what caused this increase in traffic. If you are being DDOSed you should try to figure out an anti-DDOS solution which could involve a combination of a software firewall or an appliance depending on your application and financial motivations.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.