5

We have had Nagios running on one of our servers with out any problems for a while but lately we get (Return code of 141 is out of bounds).

The load on the server raised because we gone online with our service, but its still not really high (load average max: 0.7). Before the launch everything in Nagios works fine.

See on the image, Current Load returns code 141. 2 Minutes ago the Beancounters VZ has returned 141. This happens irregular. Only HTTP & PING does not return 141, they don't relay on nrpe.

http://pic-hoster.net/view/45030/ScreenShot2012-05-28at5.31.35PM.png

I noticed that if i execute the command from my Nagios host against the problematic client, sometimes the return get lost:

root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
APT OK: 0 packages available for upgrade (0 critical updates). 
root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
root@xxx23:/usr/local/nagios/libexec# ./check_nrpe -H 123.123.123.123 -c check_apt 
APT OK: 0 packages available for upgrade (0 critical updates). 

This do not happen if i execute it directly on the client.

What i have done:

  • I increased the OpenVZ memory and CPUUnit for this container.
  • I updated to latest Nagios 3.4.1 (from source)
  • I executed the Nagios checks localy through nrpe - never got 141 back or something

I had the same issue some month ago with an other server. Haven't found the problem and reinstalled the server. Works now.

Someone with an idea?

UPDATE

I think i found it, has not happened for an hour.

SIGPIPE was a good tip, i assumed something with the system not with nagios.

I tuned the openvz configuration and limits. I will report back, if it stays stable.

0

3 Answers 3

2

We had a similar problem where one service checked via NRPE in a container returned an expected WARNING, then after some minutes the same service returned CRITICAL with the 141/SIGPIPE error. On the next check it returned WARNING then CRITICAL, then WARNING and so on.

I performed a traffic capture for the error and found Nagios issue #305 to quite precisely describe what I had observed. It seems to be caused by an unclean connection close on the NRPE server side while using SSL (SSL_shutdown()) which makes it send a TCP RST to the client which causes an aborted read and thus the SIGPIPE.

Applying the patch nrpe-ssl_shutdown-2.patch attached to the issue report to the NRPE source, rebuilding and reinstalling/restarting it seemed to stop the problem from repeating, and warnings are now reported normally without critical errors.

1

We had this problem on several occasions; it seems to be caused by the plugin dying unexpectedly.

The actions we took:

  1. Increase plugin timeout in Nagios to 120
  2. On some complex perl plugins, disabled the EPN (Add to 2nd line of script #nagios:-epn)
  3. Where check used NRPE, ensures the NRPE was using /dev/urandom so that it would never block for lack of entropy
  4. Set a reasonable command_timeout (30 sec) in nrpe.cfg
  5. Ensure Nagios server has sufficient memory/CPU to run all the checks that need to be running concurrently.

Between them, these seemed to solve the issue.

1

An exit code of 141 corresponds to signal 141 - 128 = 13. man 7 signal tells us that signal 13 is SIGPIPE (ie the process we were talking to went away). There's a bug report about this being related to OpenVZ not sending earlier signals at https://bugzilla.openvz.org/show_bug.cgi?id=1901

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .