random high cpu load on linux box

Question

It was all working fine couple of days ago (never jumped above 3.0). But now, the load average for 1 minute jumps well above 10, may hit 20, and then goes back to 5-6, maybe 3.

The htop shows me that the most cpu-consumming process is mysql, but SHOW PROCESSLIST shows nothing suspicious, and i have optimized all the queries.

The only problem that I'm worried about is disk space: i have only 4 GB left. Could that cause high cpu load? I can't imagine other problem.

sar:

10:24:52 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
10:24:53 PM       all     10.52      0.00      1.84     21.54      0.00     66.10
10:24:54 PM       all      1.28      0.00      0.26      4.28      0.00     94.19
10:24:55 PM       all      0.32      0.00      0.13      8.30      0.00     91.25
10:24:56 PM       all      6.28      0.00      1.24      3.07      0.00     89.40
10:24:57 PM       all      0.19      0.00      0.13      8.70      0.00     90.98
10:24:58 PM       all      0.59      0.00      0.20      5.14      0.00     94.08
10:24:59 PM       all      1.83      0.00      0.44      0.82      0.00     96.91
10:25:00 PM       all      1.50      0.00      0.59      1.63      0.00     96.29
10:25:01 PM       all      4.39      0.00      1.32      0.75      0.00     93.54
10:25:02 PM       all      4.88      0.00      0.95      4.63      0.00     89.54
10:25:03 PM       all      0.12      0.00      0.00      4.43      0.00     95.44
10:25:04 PM       all      0.06      0.00      0.06     16.30      0.00     83.57
10:25:05 PM       all      2.63      0.00      0.90      5.06      0.00     91.41
10:25:06 PM       all      3.44      0.00      1.10     11.34      0.00     84.13
10:25:07 PM       all      0.26      0.00      0.00     28.91      0.00     70.83
10:25:08 PM       all      1.23      0.00      0.61     20.03      0.00     78.13
10:25:09 PM       all      3.44      0.00      0.83     16.44      0.00     79.30
10:25:10 PM       all      7.79      0.00      1.47      2.27      0.00     88.47
10:25:11 PM       all      2.29      0.00      0.52      3.27      0.00     93.92
10:25:12 PM       all      0.19      0.00      0.06      8.81      0.00     90.94
10:25:13 PM       all      1.15      0.00      0.32      1.98      0.00     96.54
10:25:14 PM       all      0.76      0.00      0.38      7.38      0.00     91.48
10:25:15 PM       all      0.70      0.00      0.45      2.67      0.00     96.18
10:25:16 PM       all      1.50      0.00      0.61     30.74      0.00     67.14
10:25:17 PM       all      3.37      0.00      1.17     26.81      0.00     68.65
10:25:18 PM       all      3.27      0.00      0.96     26.83      0.00     68.93
10:25:19 PM       all     10.31      0.00      2.09     15.96      0.00     71.64
10:25:20 PM       all     10.23      0.00      2.47      5.55      0.00     81.75
10:25:21 PM       all     10.80      0.00      2.56      0.78      0.00     85.87

Answer 1

Its unable to suspect the disk space. You can enable all sets of logs in mysql, specially slow query log "log_slow_queries" will display queries that taking long time to execute, to further monitor the mysql process through logs.

There can be case like if your application gets huge requests and mysql unable to respond and waiting in the queue. So load average goes up.

You may find [1] useful.

[1] http://bit.ly/w2rPmz