Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently I noticed that there are some warnings in the nagios.log:

[1366060611] Warning: The check of service 'pt-deadlock-logger' on host 'xx' looks like it was orphaned (results never came back). I'm scheduling an immediate check of the service...

The critical problem is: after that, Nagios does not run any check anymore. As a workaround, I have to set up an event handler to restart Nagios whenever seeing this warning:

localhost.cfg

define service{
    use                     logfile-service
    host_name               localhost
    service_description     nagios_orphaned
    check_command           check_nagios_orphaned
    event_handler           restart_nagios
    contact_groups          admin
}

commands.cfg

define command {
    command_name    check_nagios_orphaned
    command_line    sudo $USER2$/check_logfiles --tag=orphaned --logfile=/usr/local/nagios/var/nagios.log --warningpattern="looks like it was orphaned"
}

define command {
    command_name    restart_nagios
    command_line    $USER1$/eventhandlers/restart_nagios.sh $SERVICESTATE$
}

restart_nagios.sh

#!/bin/bash

case "$1" in
        OK)
                ;;
        WARNING)
                /usr/bin/screen -S nagios -d -m sudo /etc/init.d/nagios restart
                ;;
        UNKNOWN)
                ;;
        CRITICAL)
                ;;
esac

exit 0

I've been trying to update the Nagios to the latest version:

# nagios -V

Nagios Core 3.5.0
Copyright (c) 2009-2011 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 03-15-2013
License: GPL

but still get this warning.

The first result when Googling is: http://support.nagios.com/wiki/index.php/Nagios_XI:FAQs#Check_Services_Being_Orphaned

but I'm sure there is only one (parent) process running:

# ps -ef | grep '/usr/local/nagios/bin/nagio[s]'
nagios    8956 15155  0 18:08 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios    8957 15155  0 18:08 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   15155     1  5 14:09 ?        00:13:47 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

Moreover, I don't see the Resource temporarily unavailable error in the log file, so the ulimit restrictions possibility can be excluded.

The embedded Perl interpreter was already disabled:

enable_embedded_perl=0
use_embedded_perl_implicitly=0

Is there any other causes?

PS: I'm running Nagios on a Xen HVM:

# virt-what 
xen
xen-hvm

UPDATE Tue Apr 16 22:07:09 ICT 2013

Search for this warning in the source code directory, I found:

# grep -lr 'looks like it was orphaned' nagios-3.5.0
/nagios-3.5.0/base/checks.o
/nagios-3.5.0/base/nagios
/nagios-3.5.0/base/checks.c

and this is the check_for_orphaned_services function:

/* check for services that never returned from a check... */
void check_for_orphaned_services(void) {
    service *temp_service = NULL;
    time_t current_time = 0L;
    time_t expected_time = 0L;


    log_debug_info(DEBUGL_FUNCTIONS, 0, "check_for_orphaned_services()\n");

    /* get the current time */
    time(&current_time);

    /* check all services... */
    for(temp_service = service_list; temp_service != NULL; temp_service = temp_service->next) {

        /* skip services that are not currently executing */
        if(temp_service->is_executing == FALSE)
            continue;

        /* determine the time at which the check results should have come in (allow 10 minutes slack time) */
        expected_time = (time_t)(temp_service->next_check + temp_service->latency + service_check_timeout + check_reaper_interval + 600);

        /* this service was supposed to have executed a while ago, but for some reason the results haven't come back in... */
        if(expected_time < current_time) {

            /* log a warning */
            logit(NSLOG_RUNTIME_WARNING, TRUE, "Warning: The check of service '%s' on host '%s' looks like it was orphaned (results never came back).  I'm scheduling an immediate check of the service...\n", temp_service->description, temp_service->host_name);

            log_debug_info(DEBUGL_CHECKS, 1, "Service '%s' on host '%s' was orphaned, so we're scheduling an immediate check...\n", temp_service->description, temp_service->host_name);

            /* decrement the number of running service checks */
            if(currently_running_service_checks > 0)
                currently_running_service_checks--;

            /* disable the executing flag */
            temp_service->is_executing = FALSE;

            /* schedule an immediate check of the service */
            schedule_service_check(temp_service, current_time, CHECK_OPTION_ORPHAN_CHECK);
            }

        }

    return;
    }

Update at: Thu Apr 18 22:32:19 ICT 2013

Just to confirm, I have edited the source code to add the value of expected_time and current_time to the log file. What I get is:

[1366294608] expected_time: 'Thu Apr 18 21:16:36 2013
', current_time: 'Thu Apr 18 21:16:48 2013
' - Warning: The check of service 'Check_MK' on host 'xx' looks like it was orphaned (results never came back).  I'm scheduling an immediate check of the service...

Re-reading the log file, I see an important message:

[1366218303] Warning: A system time change of 0d 0h 0m 1s (backwards in time) has been detected. Compensating...

Looks like Xen is the culprit.

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.