I have a NagiosXi server monitoring 631 services on 63 hosts. Every seven hours the load on the server spikes up to 20ish and then gradually falls back to near-0.
There are no cron jobs running every 7 hours.
The server has 8 cores and 2Gb RAM. The RAM is not an issue, it still sits at 1Gb free during the spikes, and upping it to 4Gb makes no difference. The server was also migrated to a new host a week or so ago with no changes.
We also have scheduled downtime on 17 of the hosts being monitored so they are only monitored during 6am-6pm Mon-Fri, this seems to make no difference to the load spikes.
Most checks are done on Windows servers, using check_wmi_plus.
During load spikes, I tend to see 5-8 instances of check_wmi_plus.pl using 2-3% cpu, and a handful of httpd processes using the same, but nothing stands out as using a lot of cpu. Those processes also roll over quite fast so they are not hung or taking an unusual long period of time. The Service Check Execution Time in NagiosXi Performance Monitor tends to peak at ~5.5s with averages around 1s.
Can anyone suggest a possible cause, or how I can further troubleshoot this?