I have very busy Linux MySQL server with huge database.
Server usually work on load average 2 or less.
The server have software raid (mdadm) that is currently rebuilding.
Several times today, MySQL get slower and load average reach 200-300-500.
In same time iostat is not big - 20-30% maximum.
Swap is currently turned off, because I thought it causes the problem.
Restart MySQL do not help. If we reboot the server, everything get back to normal for couple of hours.
Last time, instead of reboot I decided to stop MySQL and drop caches. I did in this way:
sync; echo 1 > /proc/sys/vm/drop_caches sync; echo 2 > /proc/sys/vm/drop_caches sync; echo 3 > /proc/sys/vm/drop_caches
(Yes I did three in a row)
For my surprise, when I started MySQL, everything get back to normal.
What could be the problem and can I automate the "drop caches" somehow to prevent the server of slow down at least for next couple of days.