1

I'm seeing some strange behaviour when starting a process with an init script. Most likely I am doing something stupid, but perhaps someone could let me know what might cause this behaviour:

[Background: the process is mongodb's monogod running as a config server (2.4), and I'm running on Debian 6. The process runs under user 'mongodb', and I'm issuing commands via SSH as a non-root user]

  • When the server is booted, the process appears to start normally (I can see it via ps -ef, and there is a mongodb.pid file generated).
  • If I try to stop the process via sudo /etc/init.d/mongodb stop, then the weirdness begins : I get mongodb apparently not running. (The process and the pid are still there though).
  • Of course, if I try to sudo /etc/init.d/mongodb start now I get 'failed' and an error in the log about the port already in use, which is to be expected as there is actually already an instance running.
  • If I kill -15 the process and then try sudo /etc/init.d/mongodb start I get more weirdness: In the console I get Starting database: mongodb failed! (no more info). But... the process is now running and in the logs I don't see any errors, it just looks as if the process has started normally.
  • If I try to stop this process with sudo /etc/init.d/mongodb stop I get the mongodb apparently not running.

The script in question is here.

Any ideas? Thanks!

EDIT: In case it's something mongo related, here is my mongodb.conf for the config server:

configsvr=true

dbpath=/data/configdb

logpath=/var/log/mongodb/mongodb.log
logappend=true

port = 28222
bind_ip = a_public_ip,127.0.0.1,a_private_ip

auth = true
keyFile = /srv/mongodb/keyfile

nohttpinterface = true

fork = true

Additionally: I'm seeing Starting database: mongodb failed! during the boot screen (but still there is a process seemingly running normally after boot).

1 Answer 1

1

If you have more than one instance running on that server, or something else that might be putting another value here:

PIDFILE=/var/run/$NAME.pid

That would cause the problem you are seeing here, it will be looking for the process with the ID that is in that file to determine if mongod and what PID to stop when shutting it down.

You should also take a look at the permissions on that file to make sure that the process can access it and update it when it is starting up.

Basically, we need to have separate $NAME values for each init script you are using to avoid collisions, and make sure that the right PID value is making it into that file for the service to work as expected.

If everything looks good, then I would guess you are hitting the issues described here:

https://jira.mongodb.org/browse/SERVER-7254

numactl being invoked (as it should when installed) seems to cause the incorrect PID to be written to the PID file. We're looking for feedback troubleshooting this one, so if you could add your experience (and whether you have numactl installed) that would be appreciated.

9
  • Thanks for the response. There should only be one instance running on this machine (the config server mongod) and as far as I can see that is the case. Regarding permisions on the PID file - its root/root with readonly permissions for everyone else - should this be different? If so, I wonder why I havent seen this problem on other machines. The only other thing I;m thinking is maybe the fork=true in the config - is that a potential problem? I've added the config file to the question. Thanks.
    – UpTheCreek
    Apr 23, 2013 at 17:03
  • Hmm - it seems that if I boot fresh, the process id in the PID file is different from the one reported by ps -ef. (The pid file value is slightly higher). Any ideas what could be causing this?
    – UpTheCreek
    Apr 23, 2013 at 17:06
  • I think you are hitting the bug I added to the answer, looks like a numactl related issue. If you are on a non-NUMA machine, you could take a look at the patch here: jira.mongodb.org/browse/SERVER-8774 (for a NUMA enabled machine you should make sure mongod is run with numactl).
    – Adam C
    Apr 23, 2013 at 20:20
  • Ah - it certainly seems to match those symptoms, thanks! At least I know what it is now :) I will log into jira and add what I can.
    – UpTheCreek
    Apr 24, 2013 at 6:46
  • Regarding NUMA - I'm afraid I don't know anything about this. It's a 64 bit Linode VPS instance. Is there any way I can check?
    – UpTheCreek
    Apr 24, 2013 at 8:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .