Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a master/slave MySQL configuration within our cluster (MySQL-server-5.5.9-1.rhel5 on CentOS 5 x64 boxes). Most of the time the slaving works as advertised. Every so often, our replication code shuts down the slave io-thread so it can take a backup -- starting the io-thread when completed. This causes the replication thread to reconnect. Every so often this reconnection fails with a 2026 SSL error:

110317 20:51:22 [ERROR] Slave I/O: error connecting to master 'slave@10.X.X.X:3306'
    - retry-time: 30 retries: 86400, Error_code: 2026

I notice while the Slave_IO_State is 'Connecting to master' that the mysqld CPU usage goes to 80-100%. I also notice that although we get the above message in the logs, we don't get hem every 30 seconds which I would expect give the retry-time. Nothing else in the err logs of note.

We can connect using mysql command line using the same user and password to the same master port from the slave using the same pem files and SSL. To me this doesn't seem to be a network nor a permissions issue. Nothing on the boxes has changed recently and the process of restarting the io-thread is happening every couple of hours on all of our slave nodes with 99.9% success rate.

So in the past this has just resolved itself after 1-20 minutes but recently we could not get the $@!*(@# slave to reconnect. The 2026 errors and the high load persisted during this period. We have restarted both the master and the slave mysqld. We have rebooted the slave node (like it was a Windows box) but still is not connecting. I tried to set up a fresh slave and was able to start slaving from there. We then reinstalled the broken slave and got it [finally] to connect. I just can't believe it was bit rot though.

Anyone else seen this? I've seen discussions about problems with the 3rd party SSL library but nothing specific. I got no responses to my questions on the MySQL mailing list. Any feedback would be great.


So I just strace'd it and see some wild stuff. When the CPU is maxed I see tons of the following sequences per second:

21915 22:56:58.032221 futex(0x2aab18fbf694, FUTEX_WAIT, 1, {0, 49913000}) = -1 ETIMEDOUT (Connection timed out) <0.051091>
21915 22:56:58.083452 futex(0x2aab180b3828, FUTEX_WAKE, 1) = 0 <0.000042>
21915 22:56:58.083596 clock_gettime(CLOCK_MONOTONIC, {439592, 591304847}) = 0 <0.000043>
21915 22:56:58.083754 clock_gettime(CLOCK_MONOTONIC, {439592, 591463847}) = 0 <0.000043>
21915 22:56:58.083907 clock_gettime(CLOCK_MONOTONIC, {439592, 591617847}) = 0 <0.000043>
21915 22:56:58.084063 clock_gettime(CLOCK_REALTIME, {1300834618, 84101000}) = 0 <0.000042>

This is taking minimal administrative command traffic when this is going on. I need to poke around here some more and run mysqld in --debug mode.


We are still fighting with these problems months later. We ended up turning off SSL for replication for some of databases and have not seen the issue anymore. But we'd much rather have SSL enabled and this bug fixed.

share|improve this question
1  
Not posting this as an answer, since it's more of a workaround. Have you considered something like stunnel instead? It would take SSL out of MySQL's hands but still allow you to secure your transport... –  SmallClanger May 20 '11 at 16:59
    
Thanks for the comment. Sounds like a good idea. Since our replication is controlled by code, it's non-trivial to teach a new connection mechanism. But I'll keep it in mind. Thanks. –  Gray May 23 '11 at 12:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.