Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to run memcached on a centos box and it runs for a while, but then ends up in this state:

memcached dead but subsys locked

netstat shows this:

tcp        0      0 :::11211                    :::*                        LISTEN      
udp        0      0 0.0.0.0:11211               0.0.0.0:*

ps shows this:

nobody   21983  0.0  1.8  60272 19912 ?        Ssl  16:46   0:00 memcached -d -p 11211 -u nobody -c 1024 -m 64

Anyone know what that means?

share|improve this question
up vote 44 down vote accepted

This means the service was running at one time, but has crashed.

When you start a service, it creates a "lock" file to indicate that the service is running. This helps avoid multiple instances of the service. When you stop a service, this lock file is removed.

When a running service crashes, the lock file exists but the process no longer exists. Thus, the message.

Look at the two areas /var/run/*.pid and /var/lock/subsys/*. These are expected to agree with each other. That is, if the (emtpy file) lockfile /var/lock/subsys/crond exists, then the first line of the file /var/run/crond.pid is expected to contain the PID of the process running for this service. If no such process is running, then something is wrong. If a process is indeed running (as you see) but it is not that PID, then something is probably confused.

share|improve this answer
    
Isn't it still running if it's bound to a port and listed in ps? – Nick Brosnahan Jun 16 '09 at 0:49
    
I extended my answer. Do "ls /var/lock/subsys/memcached" (I assume this file exists) and then "cat /var/run/memcached.pid" and look at the first line. The first line is the PID you should expect to see for memcached. – Eddie Jun 16 '09 at 0:55
    
Sorry, I figured out what was going on. It was still running. I launched it using "sudo /sbin/service memcached start", but looking back in my history, I got the status using "/sbin/service memcached service". PEBCAK. All is well. Thanks for your help. – Nick Brosnahan Jun 16 '09 at 1:00
5  
Ah, and running the status not as root, you may not have had read access to /var/run/memcached.pid, thus, the command assumed the subsystem was dead because it couldn't locate the correct process. – Eddie Jun 16 '09 at 1:02

While I was starting nrpe service , i was getting "nagios dead but subsys locked"

Resolution 1: In that case I tried to delete nrpe.pid under /var/run/nrpe and lock file under /var/lock/subsys - But it has not solved my problem Then I checked the logs under /var/log/messages because nagios was not creating separate log file and observed as below:

Mar 23 00:36:48 localhost kernel: readahead-collector: starting delayed service auditd Mar 23 00:36:49 localhost auditd[1273]: Started dispatcher: /sbin/audispd pid: 1275 Mar 23 00:36:49 localhost audispd: No plugins found, exiting Mar 23 00:36:49 localhost auditd[1273]: Init complete, auditd 2.0.4 listening for events (startup state enable) Mar 23 00:36:49 localhost kernel: readahead-collector: sorting Mar 23 00:36:49 localhost kernel: readahead-collector: finished Mar 23 00:38:59 localhost nrpe[1349]: Starting up daemon Mar 23 00:38:59 localhost nrpe[1349]: Cannot write to pidfile '/var/run/nrpe/nrpe.pid' - check your privileges. Mar 23 00:38:59 localhost nrpe[1349]: Bind to port 5666 on 192.168.137.2 failed: Cannot assign requested address. Mar 23 00:38:59 localhost nrpe[1349]: Cannot bind to any address.

From the log file it was clear that nagios was not able to write .pid , I observed that nrpe directory was deleted accidentally , I created it manually under /var/run After That, I again restarted nrpe but got same error, I observed the logs again:

Mar 23 00:50:26 localhost nrpe[1502]: Starting up daemon Mar 23 00:50:26 localhost nrpe[1502]: Bind to port 5666 on 192.168.137.2 failed: Cannot assign requested address. Mar 23 00:50:26 localhost nrpe[1502]: Cannot bind to any address. Mar 23 00:51:22 localhost nagios: SERVICE ALERT: localhost;HTTP;CRITICAL;HARD;4;connect to address 127.0.0.1 and port 80: Connection refused

Solution : Then I opened the /etc/nagios/nrpe.cfg file and changed the server address to nagios client IP as below(By mistake I had mentioned the nagios server address i.e 192.168.137.2): server_address=192.168.137.3

Restarted the nrpe service and it got restarted successfully

Logs output: Mar 23 00:52:13 localhost nrpe[1526]: Starting up daemon Mar 23 00:52:13 localhost nrpe[1526]: Server listening on 192.168.137.3 port 5666. Mar 23 00:52:13 localhost nrpe[1526]: Listening for connections on port 0 Mar 23 00:52:13 localhost nrpe[1526]: Allowing connections from: 192.168.137.2

share|improve this answer
    
This does not appear to address the OP's question. – Castaglia Mar 22 at 20:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.