Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a NagiosXi server monitoring 631 services on 63 hosts. Every seven hours the load on the server spikes up to 20ish and then gradually falls back to near-0.

There are no cron jobs running every 7 hours.

The server has 8 cores and 2Gb RAM. The RAM is not an issue, it still sits at 1Gb free during the spikes, and upping it to 4Gb makes no difference. The server was also migrated to a new host a week or so ago with no changes.

We also have scheduled downtime on 17 of the hosts being monitored so they are only monitored during 6am-6pm Mon-Fri, this seems to make no difference to the load spikes.

Most checks are done on Windows servers, using check_wmi_plus.

During load spikes, I tend to see 5-8 instances of check_wmi_plus.pl using 2-3% cpu, and a handful of httpd processes using the same, but nothing stands out as using a lot of cpu. Those processes also roll over quite fast so they are not hung or taking an unusual long period of time. The Service Check Execution Time in NagiosXi Performance Monitor tends to peak at ~5.5s with averages around 1s.

Can anyone suggest a possible cause, or how I can further troubleshoot this?

share|improve this question
    
Since you say it isn't a cron job then perhaps it is nagios itself. I'd look at the nagios log to see if it is restarting every 7 hours. If you are retaining state and have horribly slow disk I/O the load would spike. During the high load time run iotop -oP to see if there is a process doing excessive I/O. –  Mark Wagner Dec 4 '12 at 1:14
    
You might want to try and see if you can spread the scheduling for those windows servers, ie running server1 at 1/7 hours and the second at 2/7 and so one, basically running each check on a different hour. –  Danie Dec 4 '12 at 7:53

2 Answers 2

A high load does NOT necessarily mean that you are using high levels of CPU only it only provides the number of process at a snapshot in time that are ready to run and receive CPU time but not how much of it.

Nagios does spin off a lot of processes rapidly depending on how you have set its monitoring schedules and at times will cause a spike as it starts a lot of processes running as fast as possible, but they might not require very much CPU or go immediately into a sleep/wait state.

BTW, if you disable NOTIFICATIONS in Nagios, this does not stop it from continuing to monitor a given host or service.

share|improve this answer

lower the rhel/centos defaults prefork settings in the default /etc/httpd/conf/httpd.conf to something more realistic.

use tools like apachebuddy.pl & apachetuner.sh to do the math on memory per process fork. allow more memory for other process on the system (mysql/postgresql/php) and reduce the MaxClient and MaxRequestChild.

I experienced this after the upgrade to 2014R1.1 from 2012R2.9. not sure if the latest version of XI2014 requires more resources for the web frontend.

This morning after lowering my settings, I noticed my load spikes are smaller, and navigating through the interface doesn't give me the grey unhappy face screen using forward and back buttons in browser. does this weirdness in the interface seem similar?

One last item, I'm looking at now, is what rhel modules in this default httpd.conf file are required. I see no sense in loading default modules if not needed. This server is a PROD enterprise server at my place of business with thousands of checks, so it needs to be solid.

UPDATE:

run

# service mysqld stop

# sh /usr/local/nagiosxi/scripts/repair_databases.sh

# service mysqld start

or optimize tables while online via

# mysql -u root -p

mysql> use nagios;

list your tables

mysql> show tables;

then

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

...

mysql> use nagiosql;

list your tables

mysql> show tables;

then

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

...

do this for all tables.

If you can stop the service for the couple of minutes, then do it via nagiosxi script. if you can't until a later time... do it online, but expect the interface to be a bit slow until queries are re-ran. It maybe also beneficial to flush your query cache

mysql> FLUSH QUERY CACHE;

http://assets.nagios.com/downloads/nagiosxi/docs/Repairing_The_Nagios_XI_Database.pdf

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.