I am using tcpdump to capture UDP packets and analyze network latency between a UDP broadcaster and my server. To calculate latency I compare the source host timestamps reported in UDP application data to the tcpdump local "kernel" timestamps.
The two server clocks are synchronized to millisecond precision and a latency of up to 2 or 3 milliseconds is acceptable. My machine is a dual 6 cores Intel X5670 @2.93Ghz with 10Gb Mellanox NICs and coalescing disabled.
The problem is that I am observing greater than 10 milliseconds delay between the 2 hosts when data frequency picks up, although always well below the 10Gb bandwidth. I have 5 tcpdump jobs running at the same time as:
tcpdump -c 0 chrt -f 80 -q -i eth1 net 232.xxx.xxx.xxx and udp port 22456 -s 0 -w myfile
I had better success in the past writing my own dump program with 2 threads: one high priority network reader thread and a low priority disk writer asynchronous thread, but I'd really like to use a standard linux solution this time.
What can I do to minimize the tcpdump UDP read delay?