0

Using MCollective (2.2.3) with RabbitMQ (3.1.5) on RedHat 6.5, MCollective stops responding after a first successful ping.

When I restart RabbitMQ, it works again.

root@rabbitmqserver# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective  |  grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg  <rabbit@srv98.3.549.0>  Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg  <rabbit@srv98.3.870.0>  Q_/exchange/mcollective_directed/srv98  false
amq.gen-lwttnRCCTON9hgPg7X2zZw  <rabbit@srv98.3.1047.0> Q_/exchange/mcollective_directed/srv99  false
amq.gen-pG4o5csOGcam9mxvIQvQ7g  <rabbit@srv98.3.340.0>  Q_/exchange/mcollective_directed/srv57  false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA  <rabbit@srv98.3.545.0>  Q_/exchange/mcollective_directed/srv115 false


user@mco-client$ mco ping

srv114                        time=66.49 ms
srv57                         time=69.34 ms

---- ping statistics ----
2 replies max: 69.34 min: 66.49 avg: 67.91 
user@mco-client$  mco ping


---- ping statistics ----
No responses received

root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective  |  grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg  <rabbit@srv98.3.549.0>  Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg  <rabbit@srv98.3.870.0>  Q_/exchange/mcollective_directed/srv98  false
amq.gen-lwttnRCCTON9hgPg7X2zZw  <rabbit@srv98.3.1047.0> Q_/exchange/mcollective_directed/srv99  false
amq.gen-pG4o5csOGcam9mxvIQvQ7g  <rabbit@srv98.3.340.0>  Q_/exchange/mcollective_directed/srv57  false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA  <rabbit@srv98.3.545.0>  Q_/exchange/mcollective_directed/srv115 false

If I try using another command (such as package), it fails right after the discovery (after restarting RabbitMQ that is):

user@mco-client$ mco package status bash
Do you really want to operate on packages unfiltered? (y/n): y

 | [ >                                                             ] 0 / 2
[... stalled ...]

After a while, I start seeing this in my MCollective server logs:

E, [2014-03-18T11:51:01.332969 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.759653 #24456] ERROR -- : agents.rb:138:in `dispatch' Execution of discovery failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.760040 #24456] ERROR -- : agents.rb:139:in `dispatch' /usr/lib/ruby/gems/1.8/gems/stomp-1.2.2/lib/stomp/connection.rb:331:in `publish'
        /usr/libexec/mcollective/mcollective/connector/rabbitmq.rb:184:in `publish'
        /usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:230:in `publish'
        /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
        /usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:218:in `publish'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:129:in `reply'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:83:in `agentmsg'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:132:in `dispatch'
        /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:125:in `dispatch'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `initialize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `new'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `dispatch'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:82:in `agentmsg'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:55:in `run'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `loop'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `run'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:30:in `daemonize_runner'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:13:in `daemonize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `fork'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `daemonize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:20:in `daemonize_runner'
        /usr/sbin/mcollectived:43
E, [2014-03-18T11:56:01.336954 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:01:01.343415 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:06:01.347218 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:11:01.354069 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection

even though the machine is still listed as a consumer on the RabbitMQ server.

What could be going wrong here?

0

RabbitMQ had disk_free_limit set to 1000000000 in rabbit.app. This caused RabbitMQ to drop connections when available disk space in /var was lower than 950MB…

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.