Been having a strange problem lately and wondered if anyone had any input. I'm running a mysql replication setup, with 1 master and 3 slaves. Every 3 hours or so the binlog on the master fills up and rotates out into a new one. When this happens the master server hangs, and all the connections fill up.
To get it going I've been restarting MySQL. During the problem, disk access goes to almost 0 (just some reads of ~100 blocks according to vmstat) and I have over a gig of free RAM. MySQL is consuming minimal CPU (~20%) and the load average is close to 0. No swapping, etc..
It's like MySQL just gives up trying to accomplish any queries and as a result the connection queue fills up.
MySQL 5.1.47 CentOS 5.5
Like I said, this problem corresponds in time exactly when the master server rotates the binlog, I have expire_log_days set to 7, so it's deleting log files too and it seems specific to this deletion rather than creation of a new logfile. If I perform a manual FLUSH LOGS the server has no problem.
Would appreciate anyone's input.