Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've recently setup a munin-node on a CentOS server. All was working fine until I tried to add the apache plugin (which works fine).

For some odd reason, the mysql plugins for munin that used to work ceased to work... I'm now getting a weird error whenever I'm running the plugin with munin-run. For instance

munin-run mysql_files_tables

returns me

IPC::ShareLite store() error: Identifier removed at /usr/lib/perl5/vendor_perl/5.8.8/Cache/SharedMemoryBackend.pm line 156

but sometimes it will also return

table_open_cache.value 64

Open_files.value 58

Open_tables.value 64

Opened_tables.value 19341

but after a while it will revert to the previous error.

I do not have any knowledge about the IPC or the ShareLite library so I don't really know were to start looking. Since it is a module related to shared memory, I tried tracking down shared memory segments with ipcs without much success.

I haven't yet rebooted the machine as it is used for many projects (I'd obviously like to be able to diagnose the problem without requiring a restart if it was possible).

Has anyone faced this problem? (a quick search on google didn't present any relevant help)

Thanks for the help!

share|improve this question

1 Answer 1

up vote 0 down vote accepted

I've solved my problem, but I haven't found the exact cause/source yet, so I'm still open to hear from others.

Basically, the mysql plugins of munin are written in perl, use IPC::ShareLite.

Using strace I was able to determine the problem is that the application gets stuck on a semop on a particular semaphore id. Using ipcs and ipcrm, I was able to remove the semaphore and get the plugin running correctly again.

strace gives me the semid as the first argument passed to semop. With ipcrm -s semid, I remove the semaphore from the system, which fixes the issue.

Thus, I'm led to believe something went wrong with the semaphore (contention maybe, but I don't see from what...).

share|improve this answer
    
For a more detailed "explanation", you can check the post I wrote on my blog tomrochette.com/…. –  tomzx Oct 3 '13 at 2:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.