1

I’ve just installed PE using on Ubuntu 10.04 LTS using puppet-enterprise-2.0-ubuntu-10.04-amd64.tar.gz (downloaded from http://puppetlabs.com/misc/pe-files/) and right now the same machine is running the master, the console, and the agent roles.

I seem to have an issue with the MCollective server, which even though it seems to be starting fine, doesn’t open up port 61613 as it should.

Daemon starting fine

$ sudo /etc/init.d/pe-mcollective start
 * Starting mcollective daemon                  [ OK ]

Showing up is PS

$ ps aux | grep mcol | grep -v grep
root     25636  0.0  0.5  64680 12056 ?        S    07:21   0:00 /opt/puppet/bin/ruby /opt/puppet/sbin/mcollectived --pid /var/run/pe-mcollective.pid

Pid file contains correct value

$ sudo cat /var/run/pe-mcollective.pid
25636

However I don’t see any process listening on port 61613

$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2395/mysqld     
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      25542/memcached 
tcp        0      0 0.0.0.0:8140            0.0.0.0:*               LISTEN      25620/pe-httpd  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      10096/apache2   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2345/sshd       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2822/master     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      25620/pe-httpd

When I look at the logs I see errors in both mcollective server and client logs

$ sudo tail -f /var/log/pe-*/*
==> /var/log/pe-puppet-dashboard/mcollective_client.log <==
I, [2011-12-20T07:26:22.731870 #30944]  INFO -- : stomp.rb:79:in `on_connectfail' Connction to stomp://[email protected]:61613 failed on attempt 1646

==> /var/log/pe-mcollective/mcollective.log <==
I, [2011-12-20T07:26:24.852641 #25636]  INFO -- : stomp.rb:79:in `on_connectfail' Connction to stomp://[email protected]:61613 failed on attempt 20

When I inspect the mcollective ruby script /opt/puppet/sbin/mcollectived I can see it references the configuration file /etc/puppetlabs/mcollective/server.cfg which has the following configuration directives:

plugin.stomp.pool.size = 1
plugin.stomp.pool.host1 = puppetmaster.example.org
plugin.stomp.pool.port1 = 61613
plugin.stomp.pool.user1 = mcollective
plugin.stomp.pool.password1 = ***************
plugin.stomp.pool.ssl1 = true
plugin.stomp.base64 = true

I can resolve plugin.stomp.pool.host1, if I change plugin.stomp.pool.port1 for instance to 61614 I see it reflected in the mcollective.log errors mentioned above. I set loglevel = debug but it doesn’t give me more hints. I ran the script manually (/opt/puppet/bin/ruby /opt/puppet/sbin/mcollectived) but it doesn’t give me any clue either.

Q: How can I troubleshoot this issue further?

Improve this question

2 Answers 2

Reset to default
1

The issue was with java keystore:

pe-activemq started fine

$ sudo /etc/init.d/pe-activemq start
 * pe-activemq started

However by watching PS I noticed activemq tried to start but failed almost immediately. I modified the startup script to do some debugging:

start() {
    echo -n "Starting $APP_NAME... "
    getpid
    if [ "X$pid" = "X" ]
    then
        # original command
        # COMMAND_LINE="$CMDNICE $WRAPPER_CMD $WRAPPER_CONF wrapper.syslog.ident=$APP_NAME wrapper.pidfile=$PIDFILE wrapper.daemonize=TRUE $ANCHORPROP $IGNOREPROP $LOCKPROP"
        # custom command with deamonize=FALSE
        COMMAND_LINE="$CMDNICE $WRAPPER_CMD $WRAPPER_CONF wrapper.syslog.ident=$APP_NAME wrapper.pidfile=$PIDFILE wrapper.daemonize=FALSE $ANCHORPROP $IGNOREPROP $LOCKPROP"
        echo "executing [$COMMAND_LINE]"

Starting pe-activemq gave the following error

jvm 1    | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Transport Connector could not be registered in JMX: Failed to bind to server socket: stomp+ssl://0.0.0.0:61613 due to: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.DefaultSSLContextImpl)

It looked like an issue with the keystore. The /opt/puppet/activemq/conf/activemq-wrapper.conf configuration file gives you the following details:

set.default.ACTIVEMQ_BASE=/opt/puppet/activemq
wrapper.working.dir=/var/log/pe-activemq
[...]
# Enable SSL of the Stomp Connection (Note, this provides encryption only as per #10596)
wrapper.java.additional.7=-Djavax.net.ssl.keyStorePassword=puppet
wrapper.java.additional.8=-Djavax.net.ssl.keyStore=%ACTIVEMQ_BASE%/conf/broker.ks
# The trust store need not be present.
wrapper.java.additional.9=-Djavax.net.ssl.trustStorePassword=puppet
wrapper.java.additional.10=-Djavax.net.ssl.trustStore=%ACTIVEMQ_BASE%/conf/broker.ts

FYI /opt/puppet/activemq/conf is a symlink to /etc/puppetlabs/activemq In that directory, the broket.ts file exists, but broker.ks was missing

Based on this page (http://activemq.apache.org/how-do-i-use-ssl.html), I did the following

$ cd /etc/puppetlabs/activemq
$ sudo keytool -genkey -alias broker -keyalg RSA -keystore broker.ks
=> provided puppet password everywhere

Then activemq was starting fine

jvm 1    |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/opt/puppet/activemq/data/kahadb]
jvm 1    |  INFO | KahaDB is version 3
jvm 1    |  INFO | Recovering from the journal ...
jvm 1    |  INFO | Recovery replayed 1 operations from the journal in 0.022 seconds.
jvm 1    |  INFO | ActiveMQ 5.5.0 JMS Message Broker (localhost) is starting
jvm 1    |  INFO | For help or more information please see: http://activemq.apache.org/
jvm 1    |  INFO | Installing StaticsBroker
jvm 1    |  INFO | Starting StatisticsBroker
jvm 1    |  INFO | Listening for connections at: tcp://myserver:61616
jvm 1    |  INFO | Connector openwire Started
jvm 1    |  INFO | Listening for connections at: stomp+ssl://myserver:61613
jvm 1    |  INFO | Connector stomp+ssl Started
jvm 1    |  INFO | ActiveMQ JMS Message Broker (localhost, ID:myserver-44300-1324455724257-0:1) started
jvm 1    |  INFO | jetty-7.1.6.v20100715
jvm 1    |  INFO | ActiveMQ WebConsole initialized.
jvm 1    |  INFO | Initializing Spring FrameworkServlet 'dispatcher'
jvm 1    |  INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/camel
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/demo
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/fileserver
jvm 1    |  INFO | Started [email protected]:8161

ISSUE RESOLVED

Improve this answer
0

the daemon does not listen, the daemon connects to the middleware node, in your case puppetmaster.example.com:61613 which should have ActiveMQ running,

if you need more help please direct the question to the mcollective mailing list http://groups.google.com/group/mcollective-users

Improve this answer
1
  • I believe you missed this "right now the same machine is running the master, the console, and the agent roles.". I am testing everything on the same machine for now. I do see connection attempts to port 61613 made by the client to puppetmaster.example.com which is basically localhost, but no process listening on that port. Thanks for the link to the google group.
    – Max
    Dec 20, 2011 at 10:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .