What is the "right way" to monitor network?

Question

My production servers are kept at US east coast and some of the supporting apps are kept at Amsterdam in Europe. There is a Nagios instance running at US east coast as well which does a few port checks and a few checks via ssh.

Problem is that almost every day I observe packet drops using mtr (combo of traceroute and ping) as well as minor service issues which lasts for about 1 minute. I showed these mtr outputs to our service provider in Amsterdam but he denied any issue saying that ICMP (used by mtr) is not a reliable way to measure the drops since ICMP has lowest priority on the routers. So routers can drop ICMP but they'll be just fine for TCP.

How do I prove this to my service provider that there is indeed an issue with his service and he needs to fix it? What are the right tools and techniques for this?

Answer 1

It is difficult to definitively prove packet loss.

If this is your goal my recommended strategy would be to:

• set up host A and host B to test the network between
• implement iptables rule on each host to count number of packets entering/leaving
  • this means NO stateful tracking rule
• use iperf to do a TCP test for a period, e.g. 300 seconds
• dump the iptables on both hosts and compare packet counts

An alternative to using iptables is to look at the tx/rx packet counts of your interface on both hosts (e.g. ifconfig eth0) - make a note at the start of your test, do your transfer test (e.g. using SCP or FTP) - and then calculate if the packets sent from one host equal the packets received on the other host.

Any other technique is going to give you false information. It is true that hosts and intermediate routers will treat ICMP with low priority or maybe not respond to it at all. Often UDP packets are also treated as lower priority so a controlled iperf test using a UDP stream could give false results. And a TCP test without actually counting packets sent and packets received will never reveal much as the underlying operating system handles packet loss.

Answer 2

Maybe you could try installing smokeping and do some service checks (tcp, http, http, ...). It can do nice graphs of packet loss.

Answer 3

Product Recommendation:

Note: This is a commercial service and costs money $.

In my workplace we use a 3rd party network monitoring service called Wormly.

We use it mainly to make sure that websites are up and running, but we can also do checks on specific ports, etc.

You can get a basic account and set up some sensors to test TCP connections if ICMP is a problem.
It will produce graphs for you, which you can show to your provider.

The tests are done from several towers all over the world, and you can kindly ask the support team to set one particular tower as the primary _{(we use sydney, so that the graphs show a more realistic ping for our region)}

You can even specify certain text or a regex pattern which should be present in the TCP response, which is pretty cool.