I have two linux systems with ethernet interfaces connected directly through a cable.
Occasionally we get into flaky communications between the two devices. In these situations I run ping, and tcpdump on both sides. What I see is that tcpdump on one system sometimes shows icmp echo requests going out one interface, but tcpdump on the other end does not show it being received. The first couple times this happened we chalked it up to bad cables, but now it's happening too often to be able to blame it on cables. Also, bringing the interface down and back up or unplugging the cable and plugging it back in will fix it.
So how do I debug from here.. On the send side, where does tcpdump hook into the stack? What can happen to the packet that would cause it to fail to go out the wire after tcpdump shows it as being sent?
On the receiving side, how do I get information about ethernet frames that are silently dropped because of failed CRC checks or something like that?