MySQL replication lag increasing

Question

I have simple master-slave replication setup. Everything worked with no problems for more than a year.

Few days ago replication lag started to rise and it's still rising with no obvious reason.

Seconds_Behind_Master is now more than 4 days and 6 hours.

Both SQL and IO slave threads are working, there is no problem with communication.

SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: s1.my.pl
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.001674
          Read_Master_Log_Pos: 10948252
               Relay_Log_File: mysqld-relay-bin.000509
                Relay_Log_Pos: 63370742
        Relay_Master_Log_File: mysql-bin.001638
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 63370596
              Relay_Log_Space: 3523067396
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 371329
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
1 row in set (0.00 sec)

Show full processlist produces strange output in one row "exit mysqld_lock_tables()", I'm not sure what it means

 SHOW FULL PROCESSLIST;
+-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+
| Id  | User        | Host      | db   | Command | Time   | State                            | Info                  | Rows_sent | Rows_examined | Rows_read |
+-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+
| 808 | system user |           | NULL | Connect |    892 | Waiting for master to send event | NULL                  |         0 |             0 |         1 |
| 809 | system user |           | NULL | Connect | 371477 | exit mysqld_lock_tables()        | NULL                  |         0 |             0 |         1 |
| 810 | root        | localhost | NULL | Query   |      0 | NULL                             | SHOW FULL PROCESSLIST |         0 |             0 |       401 |
+-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+

SHOW_OPEN_TABLES command shows that no table is locked

Any hints what can I do to fix this or how to get more detailed diagnosis?

Answer 1

One of the problems with MySQL replication is the serialization of I/O in the slave. While the master performs several I/O tasks at once, they get written to a single relay log, which is then executed by the slave, hence serializing the I/O. Therefore your slave should always be capable of more I/O throughput than your master.

To see if you've actually got a I/O problem, install ioprompt (homepage).

In your case, there could be another problem, you might have run into a bug. Have you tried

stop slave;
start slave;

to see if that fixes the problem? Next thing try stopping and starting your MySQL slave process.