Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a NagiosXi server monitoring 631 services on 63 hosts. Every seven hours the load on the server spikes up to 20ish and then gradually falls back to near-0.

There are no cron jobs running every 7 hours.

The server has 8 cores and 2Gb RAM. The RAM is not an issue, it still sits at 1Gb free during the spikes, and upping it to 4Gb makes no difference. The server was also migrated to a new host a week or so ago with no changes.

We also have scheduled downtime on 17 of the hosts being monitored so they are only monitored during 6am-6pm Mon-Fri, this seems to make no difference to the load spikes.

Most checks are done on Windows servers, using check_wmi_plus.

During load spikes, I tend to see 5-8 instances of check_wmi_plus.pl using 2-3% cpu, and a handful of httpd processes using the same, but nothing stands out as using a lot of cpu. Those processes also roll over quite fast so they are not hung or taking an unusual long period of time. The Service Check Execution Time in NagiosXi Performance Monitor tends to peak at ~5.5s with averages around 1s.

Can anyone suggest a possible cause, or how I can further troubleshoot this?

share|improve this question
    
Since you say it isn't a cron job then perhaps it is nagios itself. I'd look at the nagios log to see if it is restarting every 7 hours. If you are retaining state and have horribly slow disk I/O the load would spike. During the high load time run iotop -oP to see if there is a process doing excessive I/O. –  Mark Wagner Dec 4 '12 at 1:14
    
You might want to try and see if you can spread the scheduling for those windows servers, ie running server1 at 1/7 hours and the second at 2/7 and so one, basically running each check on a different hour. –  Danie Dec 4 '12 at 7:53
add comment

1 Answer

A high load does NOT necessarily mean that you are using high levels of CPU only it only provides the number of process at a snapshot in time that are ready to run and receive CPU time but not how much of it.

Nagios does spin off a lot of processes rapidly depending on how you have set its monitoring schedules and at times will cause a spike as it starts a lot of processes running as fast as possible, but they might not require very much CPU or go immediately into a sleep/wait state.

BTW, if you disable NOTIFICATIONS in Nagios, this does not stop it from continuing to monitor a given host or service.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.