This is a Canonical Question about Monitoring Software.
Also Related: What tool do you use to monitor your servers?
I need to monitor my servers; what do I need to consider when deciding on a monitoring solution?
We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.
There are a lot of monitoring solutions out there. Everyone has their preference and each business has its own needs, so there is no correct answer. However, I can help you figure out what you might want to look for in choosing a monitoring solution.
What are Monitoring Systems For?
In general monitoring systems serve two primary purposes. The first is to collect and store data over time. For example, you might want to collect CPU utilization and graph it over time. The second purpose is to alert when things are either not responding or are not within certain thresholds. For example, you might want alerts if a certain server can't be reached by pings or if CPU utilization is above a certain percentage. There are also log monitoring systems such as Splunk but I am treating those as separate for this.
These two primary roles sometimes come in a single product, other times and more common is to have a product dedicated to each purpose.
What are the primary Components and Features in Monitoring Systems?
If you have mostly one OS in your environment or a primary OS, certain systems might have more options that others.
The configuration is generally done through a user interface or text files. The user interface option will generally be easier, but text files tend to be better for reuse and variables. So depending on your IT staff you might prefer simplicity over power.
Other features to look for are:
It is important to trust that when something goes wrong you will get the alert. This comes down to two things:
Some features to look for from the data store are:
Predefined Monitoring Templates:
Some Popular Monitoring Systems
There are a lot of monitoring systems out there. We have a list with a summary on this old question. For quick reference some that I hear the most about are:
How to Decide based on the above
The reason I can't tell you what to use is because every organization has its own needs. If you want to make the right choice you should think through all the above components and figure out what features are important to your organization. Then find a system or systems that claim to provide what you need and try them out. Some of these cost a little, a lot, or are free. Taking all of that into account you can then make your choice. From what I have used they are all far from perfect, but at least you can try to get something that fits.
It's helpful to distinguish between monitoring and alerting. Monitoring means collecting data and making graphs. Alerting means send me an SMS when a server goes down in the middle of the night.
Nagios is for alerting. Cacti and Munin are for monitoring. Other products combine the two functions. Zenoss and Zabbix are examples.
I'd start by answering some questions:
Do you need to monitor servers, network devices, applications, or all three?
Are there limitations on what methods you can use to monitor? Can you install monitoring clients like NRPE on the servers, or will you use SNMP, or maybe both?
Who will use the graphs, and who will use the alerts? What would you like the end result to look like? Does the look and feel of the interface matter (will business people be using this, or only tech staff?)
What are your resources, both in terms of time, skills and hardware? Do you have at least modest scripting ability? Do you need an out-of-the-box solution?
In my opinion, the first rule of both alerting and monitoring should be Keep it Simple! An organization can live or die on how it alerts and gathers data, and most of the time it will get complicated on its own anyway. Start with the basics and build from there.
Think about the services that your software provides, send alerts when these services fail, or when the risk of a failure of these services increases.
Service Level Agreements
The theory behind monitoring strategies is to tie monitoring and alerts to some sort of service level agreement. After all, you want to be alerted to the fact that you're losing money, not necessarily that there's a spike in the number of TCP connections to nji0019.myserver.com. There are various tools that will give you tons of alerts, define dependencies between alerts, but many of these checks aren't directly relevant to the service you provide to someone.
Breach of service
Identify the important services that you provide, such as the ability to serve a web site, and the ability to modify that web site (e.g. a CMS of some sort). Those should be checked (e.g. by monitoring that you can get the web page, and that you can). The failure of these two Services (used here with a capital S) should trigger an alert to notify you.
If it's important that the site responds within a reasonable amount of time, that too should trigger alerts. Sort of a "breach of SLA" if you will.
Usually there's an inherent risk of a Service failing, and often enough that risk is mitigated by the fact that you introduce redundancy, e.g. a second server, or a slave database, or extra network cards...
When that redundancy is lost, the Service is still fine, but the risk of the Service failing just went up.
This is the second major reason to trigger alerts; that redundancy is gone (e.g. that the second server died), or that there is an imminent danger that the risk will increase (e.g. disk only has 500Mb left, or disk trend indicates that the disk will go full in about 5 hours).
What about all those indicators?
But check_mk gives me 50-60 checks per host, are these all worthless?
No. All this doesn't mean you want to ditch the plethora of automatic checks you get with e.g. check_mk, but it means you should try to categorize each of the checks into what Service(s) might be affected if something does fail.
What Service would be affected if the /var/ partition fills up? What Service would be affected if the eth0 interface is down? ... if outbound TCP connections are blocked by some firewall? ... if the number of threads exceeds 800? ... if the database goes down?
You have 2 web servers, and a database server serving a site behind a load balancer you don't own (e.g. the ISP). The Service you provide is port 80 on the two servers, and they have enormous caches that can survive e.g. database downtime (database on a third server).
In this scenario, the complete failure of a web server would not result in the site being down. What has happened is that the redundancy is gone so that the risk of failure just went up. That should trigger an alert.
The complete failure of the database might not affect the ability to serve the site at all, because of the well tuned caches in place; This then doesn't affect the Service of serving the web site, but it might affect a different Service, namely updating the web site, or accepting orders...
Each Service would have its own level of service that designates how important it is to restore service or to avoid outages
Every time you get an alert, you should do one of the following: - change the system being monitored to fix the problem that caused the alert (e.g. replace the drive or reconfigure logrotate or something) - change the monitoring system to avoid the alert being sent out the next time that situation arises. (e.g. change the levels for "disk free" so that the disk can fill up to 90% instead of just 80%)
My own experience
I'm mostly familiar with Nagios and its verbose configuration, and have since been hooked on Check-mk's multisite. I recently learned that check_mk has this concept of Business Intelligence (since 1.11) which seems to match this thinking well. You can define that checks in nagios are part of a larger service and have rules that define the state of the "Service" as being a function of the state of many checks, aggregating to the worst or best state.
One of the most critical points companies forget when chosing a monitoring solution is that it's not all about solving immediate operational issues, it's about tomorrow's unforeseen issues! I mean, of course solving immediate issues is important, but trust me, in a lot of cases this short-sighted strategy will not guarantee a company's survival.
There are dozens of great monitoring solutions on the market. Shortlisting a small set of solutions that satisfy your requirements is a difficult and long task, moreover, finding one that fits your budget is even more difficult. The interesting part is finding one that's aligned with your present and your future. And there is no evaluation process to detect that, it is a matter of experience + intuition + a very important factor: Trust, which is not an easy thing to hack.
As a rule of thumb, search and dig for success stories of your shortlisted set of monitoring solutions, specially if it affects a company from your sector. Ask the vendor for their success stories, and even ask them for permission to speak with one of their customers. Companies that are not afraid of this show they have real relationships with their customers, and they don't hide that, and this is an extremely rare thing to find nowadays.
Zabbix, Icinga, Pandora FMS, op5, Datadog, New Relic... they all have their ups and downs, but the real issue is finding which one adapts better to your future.
If you are considering remote system monitoring, then it might be a good idea to look for the actual locations tests are performed from. Connectivity issues are not a thing of the past and if your hardware is serving a group in a specific region, you might want to make sure that your resources are available in that particular location.
Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?