How to setup monitors to check the reachability of external servers

Question

From within our production network, we make calls to other servers owned by us at different locations. Several of our services also make calls to these other servers hosted in other destinations. For instance, our mail server is in the US while our proxy server (squid proxy software running in one of the servers) is in India.

At times, the server in India would not be able to reach the US server. But the US server would be absolutely reachable from other network connections. While we suspect the data center network here, we would like to be aware of the issue as soon as it occurs. Is there a way to monitor the reachability of these external servers from our proxy server? And also, alert us when any of the servers are not reachable.

While it is possible to whip up some code with ping/curl/wget, I really wanted to know if something out there is already available to solve the problem.

Answer 1

While writing script to test connections, the scheduling, cross-referencing and reporting gets very complicated, very quickly. I'd recommend running a monitoring platform which will allow you to implement your own scripts -I use Nagios for this - its very easy to integrate with your own code.

it is possible to whip up some code with ping/curl/wget, I really wanted to know if something out there is already available to solve the problem

While nagios does have prewritten probes, the amount of effort in wrapping the functionality already available is tiny.

Answer 2

I wrote once a "fairly" extensive monitoring app, that incorporated ping, curl, app apis etc to check server statuses. So you can write a script/app yourself. It worked well initially, but without constant maintenance it will suffer and become a nuisance. (I did once receive 150 automated SMS messages during the night while on holiday in France...). Obviously if you keep it small, it will rot less.

The better choice would be to install a well supported framework like nagios or something that uses SNMP. Or use external services. I use the free mon.itor.us service for my personal stuff. Its fremium so their commercial offering may be better for suited for production. But if this is within your firewall they might not be suitable.

Answer 3

There are tons of good monitoring systems available, such as ZABBIX and nagios.

But for your case, a simple curl or wget cron job that utilizes the proxy should be easy enough to solve your immediate problem.

EDIT: To clarify, I mean that you should run the cron job somewhere in your production network. This way you will get an alarm both when your link to India goes down, or when the proxy can't reach the service you require.