15

I'm trying to set up Zenoss 4.2.0 on CentOS 6.3 to monitor a remote MySQL 5.5.25a server via IPv6. The firewall is open for the monitoring server and I can connect fine from the command line:

[root@zenoss ~]# mysql -u zenoss -p -h 2001:db8:81:2c::2
...
mysql> SELECT USER(),CURRENT_USER();
+-----------------------------------------+-----------------------------------------+
| USER()                                  | CURRENT_USER()                          |
+-----------------------------------------+-----------------------------------------+
| zenoss@2001:db8:16:bf:5054:ff:fec0:f7a5 | zenoss@2001:db8:16:bf:5054:ff:fec0:f7a5 |
+-----------------------------------------+-----------------------------------------+
1 row in set (0.09 sec)

Zenoss, however, generates an event "No performance data from plugin" whose details complain that it can't connect to the server:

MySQL Error: (2003, "Can't connect to MySQL server on '2001:db8:81:2c::2' (-9)")

As far as I know, -9 isn't even a valid errno. And of course it's impossible to Google a negative number.

enter image description here

I have checked zMySqlUsername and zMySqlPassword - more than once - and they have the correct values.

I have also tried entering the IPv6 address with brackets, but MySQL doesn't like that at all, either within Zenoss or on the command line.

What is the cause of this problem?

3
  • If all else fails can't you drop back to IPv4? Aug 5, 2012 at 12:50
  • @JohnGardeniers Sometimes. But many of the machines to be monitored don't have global IPv4 addresses, so a Zenoss proxy would be required. Among other things, I'm trying to move away from that. Aug 5, 2012 at 20:32
  • Ok, I just thought it might be an option, especially with IPv6 being so incomplete or imperfect in a lot of products. Aug 6, 2012 at 0:37

3 Answers 3

11

I finally gave up and went to debug this one myself.

Based on @SelivanovPavel's answer I turned up debugging on zencommand and waited, and sure enough, the ZenPack was failing.

2012-08-16 18:16:14,092 INFO zen.zencommand: Datasource MySQL/mysql command: /opt/zenoss/ZenPacks/ZenPacks.zenoss.MySqlMonitor-2.2.0-py2.7.egg/ZenPacks/zenoss/MySqlMonitor/libexec/check_mysql_stats.py -H 2001:db8:81:2c::2 -p 3306 -u zenoss -w 'password' -g
2012-08-16 18:16:14,100 DEBUG zen.zencommand: Running /opt/zenoss/ZenPacks/ZenPacks.zenoss.MySqlMonitor-2.2.0-py2.7.egg/ZenPacks/zenoss/MySqlMonitor/libexec/check_mysql_stats.py
2012-08-16 18:16:14,544 DEBUG zen.zencommand: Datasource: mysql Received exit code: 1 Output: 'MySQL Error: (2003, "Can\'t connect to MySQL server on \'2001:db8:81:2c::2\' (-9)")\n'
2012-08-16 18:16:14,545 DEBUG zen.zencommand: Process MySQL/mysql stopped (1), 0.43 seconds elapsed 

So I dug into the ZenPack and found out it was importing (an apparently old version of) pymysql from /opt/zenoss/lib/python.

On testing from the python command line I discovered where the exception was being thrown from:

>>> sys.path.insert(0, "/opt/zenoss/lib/python");
>>> import pymysql
>>> pymysql.install_as_MySQLdb()
>>> import MySQLdb
>>> self.conn = MySQLdb.connect(host="2001:db8:81:2c::2", port=3306, db='', user='zenoss', passwd='password')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/zenoss/lib/python/pymysql/__init__.py", line 93, in Connect
    return Connection(*args, **kwargs)
  File "/opt/zenoss/lib/python/pymysql/connections.py", line 504, in __init__
    self._connect()
  File "/opt/zenoss/lib/python/pymysql/connections.py", line 673, in _connect
    raise OperationalError(2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e.args[0]))
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '2001:db8:81:2c::2' (-9)")

And on inspecting connections.py in that general vicinity I discovered to my horror that it was attempting to open an AF_INET socket, and there was no code anywhere to open an AF_INET6 socket. Boom, instant fail.

The current version of pymysql also seems to contain this deficiency; no IPv6 support whatsoever.

So the "answer" is I'm going to have to fix pymysql. Not how I wanted to spend my afternoon.

This bit of nasty hackery gets things working (though you need Python 2.6). Open up /opt/zenoss/lib/python/pymysql/connections.py and search for AF_INET around line 660. Then make the following change:

                 if DEBUG: print 'connected using unix_socket'
             else:
-                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-                t = sock.gettimeout()
-                sock.settimeout(self.connect_timeout)
-                sock.connect((self.host, self.port))
-                sock.settimeout(t)
+                sock = socket.create_connection((self.host, self.port), self.connect_timeout)
                 self.host_info = "socket %s:%d" % (self.host, self.port)
                 if DEBUG: print 'connected using socket'

This has since been fixed in upstream pymysql and should be available in a future release.

5
+250

Check, are there any connection attempts:

tshark -i br200 -f "host 2001:db8:81:2c::2"

tshark is console version of packet capturing program Wireshark.

If zenoss service user is not root - try to connect to mysql from his shell:

su zenoss
mysql ...

What about Zenoss logs(Settings > Daemons)? Try to increase logs verbosity(set logseverity = 30) and see what happens.

This doc may be useful: Troubleshooting_Zenoss

1
  • You get the bounty because you came closer than the other guy to the source of the problem. Thanks. :) Aug 16, 2012 at 18:43
3

Try putting it in brackets [2001:470:...] or ipv6:[]. A great many parsers can't differentiate between a text entry and a v6 address.

4
  • 1
    Been there, done that. MySQL, at least, wants the IP address without brackets. Aug 4, 2012 at 22:00
  • 2
    That error code may be from the libraries that Zenoss is using rather than from mysql itself. It's written mostly in Python if I'm remembering correctly, so that may be a place to look for a hint.
    – rnxrx
    Aug 4, 2012 at 22:24
  • 1
    If I have to go digging into the source code, I'll end up answering my own question. Again. :) Aug 4, 2012 at 22:29
  • 3
    Isn't that the beauty of open source, though? We can all fix our own bugs <kidding>. Hey - another thought, though. What if you set up a standard hostname that only resolved to a AAAA and then used that hostname instead of the raw IP?
    – rnxrx
    Aug 5, 2012 at 1:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .