Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have had Nagios running on one of our servers with out any problems for a while but lately we get (Return code of 141 is out of bounds).

The load on the server raised because we gone online with our service, but its still not really high (load average max: 0.7). Before the launch everything in Nagios works fine.

See on the image, Current Load returns code 141. 2 Minutes ago the Beancounters VZ has returned 141. This happens irregular. Only HTTP & PING does not return 141, they don't relay on nrpe.

http://pic-hoster.net/view/45030/ScreenShot2012-05-28at5.31.35PM.png

I noticed that if i execute the command from my Nagios host against the problematic client, sometimes the return get lost:

root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
APT OK: 0 packages available for upgrade (0 critical updates). 
root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
APT OK: 0 packages available for upgrade (0 critical updates). 

This do not happen if i execute it directly on the client.

What i have done:

  • I increased the OpenVZ memory and CPUUnit for this container.
  • I updated to latest Nagios 3.4.1 (from source)
  • I executed the Nagios checks localy through nrpe - never got 141 back or something

I had the same issue some month ago with an other server. Haven't found the problem and reinstalled the server. Works now.

Someone with an idea?

UPDATE

I think i found it, has not happened for an hour.

SIGPIPE was a good tip, i assumed something with the system not with nagios.

I tuned the openvz configuration and limits. I will report back, if it stays stable.

share|improve this question
    
Did it stay stable? –  sendmoreinfo Jul 18 '12 at 20:20

2 Answers 2

We had this problem on several occasions; it seems to be caused by the plugin dying unexpectedly.

The actions we took:

  1. Increase plugin timeout in Nagios to 120
  2. On some complex perl plugins, disabled the EPN (Add to 2nd line of script #nagios:-epn)
  3. Where check used NRPE, ensures the NRPE was using /dev/urandom so that it would never block for lack of entropy
  4. Set a reasonable command_timeout (30 sec) in nrpe.cfg
  5. Ensure Nagios server has sufficient memory/CPU to run all the checks that need to be running concurrently.

Between them, these seemed to solve the issue.

share|improve this answer

An exit code of 141 corresponds to signal 141 - 128 = 13. man 7 signal tells us that signal 13 is SIGPIPE (ie the process we were talking to went away). There's a bug report about this being related to OpenVZ not sending earlier signals at https://bugzilla.openvz.org/show_bug.cgi?id=1901

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.