How to collect concise traffic statistics on linux?

Question

I need to have statistics divided by time periods, like this:

time_1 - time_1 + 5mins
from_ip1 > to_ip2  total_packages_size1
from_ip3 > to_ip4  total_packages_size2

time_1 + 5mins - time_1 + 10mins
from_ip1 > to_ip2  total_packages_size3
from_ip5 > to_ip6  total_packages_size4
...

I looked at tcpdump, and I know how to get not summarized statistics with it. But I don't really want to make a tool which processes tcpdump logs and then clears them. There should be some ready-to-use tool for this standard problem.

Answer 1

I tend to do this with accounting rules in iptables. I create a custom auditing rule

iptables -N GKT-AUDIT

I then send all traffic coming through, in this case, the OUTPUT chain, through that rule before anything else

iptables -I OUTPUT 1 -j GKT-AUDIT

I then introduce rules with no targets into the GKT-AUDIT chain

iptables -A GKT-AUDIT -s 5.5.5.1 -d 7.7.7.2
iptables -A GKT-AUDIT -s 5.5.5.3 -d 7.7.7.4

and so on. Since those rules have no targets, matching packets don't terminate but continue through the GKT-AUDIT chain, then fall out the end and return to the OUTPUT chain, so the rules have no effect on data flow. But the packets do increment the packet counts of the matching rules on their way through, so I can slice and dice my traffic any way I want from a statistical basis. I then usually collect them with a munin plugin, and have munin graph them, but that's a refinement that may or may not suit you.

The nice thing about using iptables is that it can slice the traffic so many different ways, so I can be very precise about what traffic any particular rule matches: if I want to count only packets from a certain address, to a certain other address, but only on a certain source port range, and then only if it's part of a connection that has transferred in excess of 10MB, I can do that.

Answer 2

Don't know if this will help you, but maybe it's worth looking into.

You can import raw tcpdump files in wireshark, thats a pretty nice tool. Not sure if it's possible to use the summarized files though.

Regards, jgr

Answer 3

What's your backend network infrastructure? Netflow was pretty much built for exactly the use case you're describing. It was developed by Cisco, but is available on most higher-end switches.

If you have to do this on the host level, this sounds like a job for ntop. I don't think its reports get as incremental as you want, but you can probably tie into its Python API to get what you want.