I have a server that experienced an inexplicable severe slowdown for 12+ hours, after which it fully recovered. This server is running mysql 5.5, redis, mongodb, and apache http on CentOS 5.4 with gobs of memory.

During the slowdown, the most severely affected service was mysql, which would take 2 minutes to run a simple query that normally takes under a second. After the slowdown concluded, the query would run normally again within a second. All other services were only minimally affected.

I ran the standard tests on mysql:

  • Show the full process list
  • Explained queries
  • Logged slow queries
  • Analyzed and optimized tables
  • Checked server CPU and memory utilization
  • Checked IO wait states
  • Tested HDD access times
  • Tested a second mysql instance compiled from scratch (failed, slow)
  • Restarted mysql
  • Restarted CentOS

Nothing abnormal appeared for any of these tests, although mysql showed CPU utilization at max 500% of 12 cores.

The simple SQL query contained a single join. If I removed the join then the statement ran immediately.

I feel that an invisible and untraceable slowdown like this must indicate the kernel is busted.

What else could I have examined to find the cause? Because this is the second time this server has had this problem.

  • In addition to MySQL ones, what other data have you collected during the problem and/or as a "monitoring" activity spanning a wider timeframe? Can you provide some graphs? I mean.... something more detailed than your "text-based" bullet list :-) – Damiano Verzulli May 16 '17 at 5:43
  • There could be any changes in configuration that will applied after or during recovery? – Alexander Tolkachev May 16 '17 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.