I have a server that experienced an inexplicable severe slowdown for 12+ hours, after which it fully recovered. This server is running mysql 5.5, redis, mongodb, and apache http on CentOS 5.4 with gobs of memory.
During the slowdown, the most severely affected service was mysql, which would take 2 minutes to run a simple query that normally takes under a second. After the slowdown concluded, the query would run normally again within a second. All other services were only minimally affected.
I ran the standard tests on mysql:
- Show the full process list
- Explained queries
- Logged slow queries
- Analyzed and optimized tables
- Checked server CPU and memory utilization
- Checked IO wait states
- Tested HDD access times
- Tested a second mysql instance compiled from scratch (failed, slow)
- Restarted mysql
- Restarted CentOS
Nothing abnormal appeared for any of these tests, although mysql showed CPU utilization at max 500% of 12 cores.
The simple SQL query contained a single join. If I removed the join then the statement ran immediately.
I feel that an invisible and untraceable slowdown like this must indicate the kernel is busted.
What else could I have examined to find the cause? Because this is the second time this server has had this problem.
