How can I find out what is congesting my network connection?

Question

I am asking because there has been a couple of times when our server's network connection was saturated. I want to get an idea as to what is causing the network load.

My first idea was to make an iptables log rule. E.g.:

/sbin/iptables -A INPUT -m mport --dports 20,80,53 -j LOG --log-prefix "Audit: "

But this doesn't work very well since I would have to parse to the log and collect stats. Preferably, I'd like something that would answer the following questions:

1. How many TCP connections are active for each listening port
2. How many UDP packets were received on a particular port in a set intreval
3. Which IP address has made the most connections to my server in the last x hours

Answer 1

If you have a redhat/centos/fedora based distro the package iftop is available in the EPEL repositories, which gives a per connection breakdown, which will enable you to determine whether its one particular host that is causing the problem.

You probably want the command "iftop -P -N" which displays the ports in addition to the hosts as in the screenshot below.

Some details

Also appears in the deb/ubuntu/gentoo repos somewhere...
sudo aptitude install iftop
yum install iftop
emerge iftop

With regard to your 3 questions;

1) How many TCP connections are active for each listening port

netstat -ant | egrep 'ESTABLISHED' | awk '{print $4}' | \
awk -F: '{print $NF}' | sort -n | uniq -c


      5 22
      1 389
      1 766
      1 778
      1 812
      1 838
      1 4369

that is a pretty basic off the top of my head, for chewing the output of netstat, you can change the grep to add or remove various states ...

2) How many UDP packets were received on a particular port in a set intreval 3) Which IP address has made the most connections to my server in the last x hours

In both these cases, you are going to need to collect data, like you suggested the simplest way is some sort of iptables -j LOG rule. The first answer in this question gives some examples of that;
http://stackoverflow.com/questions/349576/linux-retrieve-per-interface-sent-received-packet-counters-ethernet-ipv4-ipv

For more general visualization tools , you might consider iptraf or ntop which generate more comprehensive reports and data.

There is a fairly comprehensive list of network monitoring tools here;
http://www.ubuntugeek.com/bandwidth-monitoring-tools-for-linux.html

Answer 2

Run a packet capture and then use Wireshark to analyze it, this should give you a good idea. There are plenty of tutorials out there to help you figure out how this is accomplished. If the server happens to have a GUI, etherape can help you visualize what's going on pretty easily.

Answer 3

You can make use or recent and/or limit iptables modules. Have a look at this page.