on one server I'm currently experiencing strange application behavior that seems all related to network issues.

Example of problems:

  • applicationA detects a lost connection to serverB, but serverB still see the connection as opened
  • applicationA does soap webservice calls to serverC, serverC never receives the soap requests but applicationA doesn't do a timeout nor error.

How would you go about investigating those kind of issues ? Concerning first problem, I know that there should always have 3 connections open towards serverB so I simply have a monitoring in place that do something like:

netstat | grep IP_serverB

This is ok to detect when connections are lost but I would like to investigate why those connections are lost. How would you do that ?

Which tools do you need to install ?

I know a partner of us which seems to monitor all traffic going through its server and is able to provide us with packet traces files (.pcap file).

How would you set that up ? Would it help to investigate problems ?


Odds are the pcaps are being created by tcpdump, which I cant recommend enough. I cant count the number of times its helped me figure out network problems. So I would set up tcpdump on each server, and limit the capturing to just server A, B or C as appropriate:

tcpdump -i any -v "host <ip of server>" -s0 -w output.pcap

Note that using the host parameter means that traffic for that host (whether incoming or outgoing) will be captured.

Also, note that output.pcap may not end up where you think it should. (in the above example the current directory) This is because some versions of tcpdump will chroot before running. If you dont find the file you expect, check in /var/lib/tcpdump

run the tcpdumps until you are satisfied you have captured an example of each situation. If you want you can further limit the packet capture by adding additional conditions e.g.

host <ip of server> and port 80

Once you have your pcap file, you can load it into wireshark and hopefully figure out where things are going wrong.

  • To clarify, the tcpdump command will record the data into a pcap (standard) file. Thereafter, you can read that file with Wireshark which provides a nifty UI making it easier to debug. – dmourati Oct 3 '13 at 21:20

Such issues may occur if you are tunnelling connections or having a NAT somewhere on the way, which works one way but not properly doing NAT back. Routing may be an issue as well.

The simplest way to start with is to check the routing on all 3 machines. You can do that with:netstat -rn

I don't know your specific setup but you should be using proper interfaces and gateways, pay attention to netmasks as well.

Then, have a look at the NAT table:

iptables -L -n -t nat

Sometimes firewall may be an issue so worth checking those too:

iptables -L -n

You haven't specified the protocol that you use (UDP/TCP) so I'm assuming TCP. TCP connections have several states in which they can be. You can have Cacti monitor those for you in a graphical way, have a look here on Cacti forums, how to set it up: http://forums.cacti.net/viewtopic.php?f=12&t=12787

On top of that, you can add ip_conntrack module ( modprobe nf_conntrack ) or just compile it into your kernel (harder, but maybe your default Linux kernel already supports it). Check with:

cat /proc/net/ip_conntrack

You will see a connections list along with their status. You can use grep to filter out what you don't need or just look for what is of your interest. You can have it run every minute (from crontab) and dump the connection status into a file, like this:

grep IP_serverB /proc/net/ip_conntrack > `date +%Y%m%d_%H:%M`

Which will put all the connections to/from IP_serverB into a file named: 20131003_23:11 into the current directory.

That way you can see what happens on both of the machines (or 3 of them, if you let it run on 3 of them). Wikipedia has a nice graph illustrating each TCP state here: http://upload.wikimedia.org/wikipedia/commons/thumb/a/a2/Tcp_state_diagram_fixed.svg/796px-Tcp_state_diagram_fixed.svg.png

You can also setup tcpdump to log your tcp connections to/from given machines in this way:

tcpdump -i <network_interface> host <IP> and ip proto \\tcp and port <port> -l > tcpdump_logfile &

Or if you want to look at it real time AND log it into a file at the same time, do:

tcpdump -i <network_interface> host <IP> and ip proto \\tcp and port <port> -l | tee tcpdump_logfile

With extra -v(vv) parameters you can increase verbosity. Tcpdump will log all the details of TCP connections (without the data itself). This should also help you diagnose, what happens on the wire.

Ettercap/wireshark/tshark could be your friend if you wanted to actually see the data within TCP packets.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.