Lots of dropped packages when tcpdumping on busy interface

Question

My challenge

I need to do tcpdumping of a lot of data - actually from 2 interfaces left in promiscuous mode that are able to see a lot of traffic.

To sum it up

• Log all traffic in promiscuous mode from 2 interfaces
• Those interfaces are not assigned an IP address
• pcap files must be rotated per ~1G
• When 10 TB of files are stored, start truncating the oldest

What I currently do

Right now I use tcpdump like this:

ifconfig ethX promisc
ifconfig ethX promisc
tcpdump -n -C 1000 -z /data/compress.sh -i any -w /data/livedump/capture.pcap $FILTER

The $FILTER contains src/dst filters so that I can use -i any. The reason for this is, that I have two interfaces and I would like to run the dump in a single thread rather than two.

compress.sh takes care of assigning tar to another CPU core, compress the data, give it a reasonable filename and move it to an archive location.

I cannot specify two interfaces, thus I have chosen to use filters and dump from any interface.

Right now, I do not do any housekeeping, but I plan on monitoring disk and when I have 100G left I will start wiping the oldest files - this should be fine.

And now; my problem

I see dropped packets. This is from a dump that has been running for a few hours and collected roughly 250 gigs of pcap files:

430083369 packets captured
430115470 packets received by filter
32057 packets dropped by kernel  <-- This is my concern

How can I avoid so many packets being dropped?

These things I did already try or look at

Changed the value of /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_default which did indeed help - actually it took care of just around half of the dropped packets.

I have also looked at gulp - the problem with gulp is, that it does not support multiple interfaces in one process and it gets angry if the interface does not have an IP address. Unfortunately, that is a deal breaker in my case.

Next problem is, that when the traffic flows though a pipe, I cannot get the automatic rotation going. Getting one huge 10 TB file is not very efficient and I don't have a machine with 10TB+ RAM that I can run wireshark on, so that's out.

Do you have any suggestions? Maybe even a better way of doing my traffic dump altogether.

Answer 1

tcpdump stores incoming data in a ring buffer. If the buffer overflows before tcpdump processes its contents, then you lose packets.

The default ring buffer size is probably 2048 (2MiB).

To increase the buffer size, add the -B option:

tcpdump -B 4096 ...

You should also try using faster disk storage.

Answer 2

I have ended up finding a solution that is to live with. The dropped packages has been decreased from .0047% to .00013% - which doesn't seem like much at first, but when we are talking millions of packets, it is quite a lot.

The solution consisted of several things. One was to change the ring buffer size as suggested by Michael Hampton.

Also, I created a ramfs and did live dumping to that, rewrote my compress script to take care of moving the dumps from ramfs to disk. This only decreased the amount very little, but enough to be notable - even though all testing and benchmarking of the disk shows, that the disk should not be the bottleneck. I guess the accesstime is very important here.

Disabling hyper threading also did more than you'd thought.