Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to set up Zenoss 4.2.0 on CentOS 6.3 to monitor a remote MySQL 5.5.25a server via IPv6. The firewall is open for the monitoring server and I can connect fine from the command line:

[root@zenoss ~]# mysql -u zenoss -p -h 2001:db8:81:2c::2
...
mysql> SELECT USER(),CURRENT_USER();
+-----------------------------------------+-----------------------------------------+
| USER()                                  | CURRENT_USER()                          |
+-----------------------------------------+-----------------------------------------+
| zenoss@2001:db8:16:bf:5054:ff:fec0:f7a5 | zenoss@2001:db8:16:bf:5054:ff:fec0:f7a5 |
+-----------------------------------------+-----------------------------------------+
1 row in set (0.09 sec)

Zenoss, however, generates an event "No performance data from plugin" whose details complain that it can't connect to the server:

MySQL Error: (2003, "Can't connect to MySQL server on '2001:db8:81:2c::2' (-9)")

As far as I know, -9 isn't even a valid errno. And of course it's impossible to Google a negative number.

enter image description here

I have checked zMySqlUsername and zMySqlPassword - more than once - and they have the correct values.

I have also tried entering the IPv6 address with brackets, but MySQL doesn't like that at all, either within Zenoss or on the command line.

What is the cause of this problem?

share|improve this question
    
If all else fails can't you drop back to IPv4? –  John Gardeniers Aug 5 '12 at 12:50
    
@JohnGardeniers Sometimes. But many of the machines to be monitored don't have global IPv4 addresses, so a Zenoss proxy would be required. Among other things, I'm trying to move away from that. –  Michael Hampton Aug 5 '12 at 20:32
    
Ok, I just thought it might be an option, especially with IPv6 being so incomplete or imperfect in a lot of products. –  John Gardeniers Aug 6 '12 at 0:37
add comment

3 Answers 3

up vote 10 down vote accepted

I finally gave up and went to debug this one myself.

Based on @SelivanovPavel's answer I turned up debugging on zencommand and waited, and sure enough, the ZenPack was failing.

2012-08-16 18:16:14,092 INFO zen.zencommand: Datasource MySQL/mysql command: /opt/zenoss/ZenPacks/ZenPacks.zenoss.MySqlMonitor-2.2.0-py2.7.egg/ZenPacks/zenoss/MySqlMonitor/libexec/check_mysql_stats.py -H 2001:db8:81:2c::2 -p 3306 -u zenoss -w 'password' -g
2012-08-16 18:16:14,100 DEBUG zen.zencommand: Running /opt/zenoss/ZenPacks/ZenPacks.zenoss.MySqlMonitor-2.2.0-py2.7.egg/ZenPacks/zenoss/MySqlMonitor/libexec/check_mysql_stats.py
2012-08-16 18:16:14,544 DEBUG zen.zencommand: Datasource: mysql Received exit code: 1 Output: 'MySQL Error: (2003, "Can\'t connect to MySQL server on \'2001:db8:81:2c::2\' (-9)")\n'
2012-08-16 18:16:14,545 DEBUG zen.zencommand: Process MySQL/mysql stopped (1), 0.43 seconds elapsed 

So I dug into the ZenPack and found out it was importing (an apparently old version of) pymysql from /opt/zenoss/lib/python.

On testing from the python command line I discovered where the exception was being thrown from:

>>> sys.path.insert(0, "/opt/zenoss/lib/python");
>>> import pymysql
>>> pymysql.install_as_MySQLdb()
>>> import MySQLdb
>>> self.conn = MySQLdb.connect(host="2001:db8:81:2c::2", port=3306, db='', user='zenoss', passwd='password')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/zenoss/lib/python/pymysql/__init__.py", line 93, in Connect
    return Connection(*args, **kwargs)
  File "/opt/zenoss/lib/python/pymysql/connections.py", line 504, in __init__
    self._connect()
  File "/opt/zenoss/lib/python/pymysql/connections.py", line 673, in _connect
    raise OperationalError(2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e.args[0]))
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '2001:db8:81:2c::2' (-9)")

And on inspecting connections.py in that general vicinity I discovered to my horror that it was attempting to open an AF_INET socket, and there was no code anywhere to open an AF_INET6 socket. Boom, instant fail.

The current version of pymysql also seems to contain this deficiency; no IPv6 support whatsoever.

So the "answer" is I'm going to have to fix pymysql. Not how I wanted to spend my afternoon.

This bit of nasty hackery gets things working (though you need Python 2.6). Open up /opt/zenoss/lib/python/pymysql/connections.py and search for AF_INET around line 660. Then make the following change:

                 if DEBUG: print 'connected using unix_socket'
             else:
-                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-                t = sock.gettimeout()
-                sock.settimeout(self.connect_timeout)
-                sock.connect((self.host, self.port))
-                sock.settimeout(t)
+                sock = socket.create_connection((self.host, self.port), self.connect_timeout)
                 self.host_info = "socket %s:%d" % (self.host, self.port)
                 if DEBUG: print 'connected using socket'

This has since been fixed in upstream pymysql and should be available in a future release.

share|improve this answer
add comment

Check, are there any connection attempts:

tshark -i br200 -f "host 2001:db8:81:2c::2"

tshark is console version of packet capturing program Wireshark.

If zenoss service user is not root - try to connect to mysql from his shell:

su zenoss
mysql ...

What about Zenoss logs(Settings > Daemons)? Try to increase logs verbosity(set logseverity = 30) and see what happens.

This doc may be useful: Troubleshooting_Zenoss

share|improve this answer
    
You get the bounty because you came closer than the other guy to the source of the problem. Thanks. :) –  Michael Hampton Aug 16 '12 at 18:43
add comment

Try putting it in brackets [2001:470:...] or ipv6:[]. A great many parsers can't differentiate between a text entry and a v6 address.

share|improve this answer
    
Been there, done that. MySQL, at least, wants the IP address without brackets. –  Michael Hampton Aug 4 '12 at 22:00
1  
That error code may be from the libraries that Zenoss is using rather than from mysql itself. It's written mostly in Python if I'm remembering correctly, so that may be a place to look for a hint. –  rnxrx Aug 4 '12 at 22:24
    
If I have to go digging into the source code, I'll end up answering my own question. Again. :) –  Michael Hampton Aug 4 '12 at 22:29
2  
Isn't that the beauty of open source, though? We can all fix our own bugs <kidding>. Hey - another thought, though. What if you set up a standard hostname that only resolved to a AAAA and then used that hostname instead of the raw IP? –  rnxrx Aug 5 '12 at 1:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.