4

How to make a TCP dump where it is guaranteed that all the packets that really pass through the network are captured, and nothing is missed?

Details: We have an issue with 3rd party vendor who provides a solution on top of SCTP stack, which he also implements.
Under quite high throughput (52 000 messages/sec, average message size is 500 bytes) the SCTP link breaks.
We believe that the bug is in the vendor SCTP stack.
But the vendor says, this happens because SCTP stack sends a message, doesn't receive ACK on it, sends a number of retransmits, doesn't receive ACKs on them as well and closes the SCTP link.
So the vendor says, this is the network which is guilty, because it loses packets.

In the TCP dumps on both sides, client and server we see that the original messages reaches the server and see that the server doesn't answer with ACK. But the vendor says that TCP dumps are not reliable, that when capturing a TCP dump, some packets could be not captured, because libpcap library works only within one hardware thread, its power can be not enough to log all the packets.

Technical data: 52 000 messages/sec, average message size is 500 bytes, so 26 MB/sec in total, 4 SCTP links are used.
Hardware: CPU E5-2670, 2.6 GHz, 8 HW threads
Network: 10 GBit, the traffic is between HP blades, which are located in one rack.
RHEL 6.

Improve this question

1 Answer 1

Reset to default
8

At your amount of traffic, I would claim that libpcap should have no trouble with dropped packets unless you have a particularly inefficient setup. If you are using tcpdump for capturing, it will report the amount of dropped packets in its final output line. If you see dropped packets, you might want to increase tcpdump's buffer size by supplying the -B option to set a value considerably higher than the default 2 MB.

Nevertheless, you might want to look at PF_RING:

Who needs PF_RING™?

Basically everyone who has to handle many packets per second. The term ‘many’ changes according to the hardware you use for traffic analysis. It can range from 80k pkt/sec on a 1,2GHz ARM to 14M pkt/sec and above on a low-end 2,5GHz Xeon. PF_RING™ not only enables you to capture packets faster, it also captures packets more efficiently preserving CPU cycles. Just to give you some figures you can see how fast nProbe, a NetFlow v5/v9 probe, can go using PF_RING™, or have a look at the tables below.

10 Gigabit tests performed on a Core2Duo 1.86 GHz and a low-end Xeon 2,5 Ghz

                                    ixgbe 
            Application                                 Rate 
pfcount (RX, with PF_RING™ DNA)        11 Mpps (Core2Duo), 14.8 Mpps (Xeon) 
pfsend (TX, with PF_RING™ DNA)         11 Mpps (Core2Duo), 14.8 Mpps (Xeon)

The PF_RING user guide explains how to compile and configure PF_RING-enabled libpcap libraries, if you insist on using libpcap applications for packet capture.

Improve this answer
1
  • Yes, we do have packets dropped during tcpdump. The following doesn't help: Increasing buffer size (-B), setting high priority to tcpdump (nice), write into RAM disk, using gulp instead of tcpdump, increasing /proc/sys/net/core/netdev_max_backlog, /proc/sys/net/core/netdev_budget, /proc/sys/net/core/rmem_max, /proc/sys/net/core/rmem_default, disabling hyperthreading. We still see dropped packets. Thanks for pointing to PF_RING. We will try it
    – Neighbour
    Jul 4, 2014 at 13:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .