Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Since some time days ago, one of my servers started crashing. Currently it is crashing almost every day. Sometimes, over once a day. Only a restart can solve.

The last time the problem happend, the Server Load was over 1.000, but the server was not slow, But I could not access websites that make requests to mysql.

According to the command iostat -dx 1 there was no data being written to the disks. That's is not normal. Some websites on the server demands a lot of data writing. So all that I can imagine is a disk problem.

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    28.00  0.00 67.00     0.00   760.00    11.34     0.17    2.46   0.04   0.30
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00    28.00  0.00 67.00     0.00   760.00    11.34     0.17    2.46   0.04   0.30
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   665.00  0.00 47.00     0.00  5696.00   121.19     0.24    5.15   0.17   0.80
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00   665.00  0.00 47.00     0.00  5696.00   121.19     0.24    5.15   0.17   0.80
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

I analyzed the iostat data for some seconds, and the %util value did not change over 0,80% any time. What is not normal, due to the websites that I host.

When the server is working fine SDA and SDA2 %util values area almost all the time over 30%

And this is top command data, while the server was facing problems: Notice that 97.1%sy is a very high value.

top - 15:47:05 up 1 day, 17:14,  1 user,  load average: 1492.16, 979.45, 435.51
Tasks: 1808 total, 1501 running, 307 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.9%us, 97.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12290048k total, 12194552k used,    95496k free,   804172k buffers
Swap:  1052248k total,     5092k used,  1047156k free,  8652196k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                            
 8795 nobody    25   0  205m 8344 2176 R  2.0  0.1   0:05.43 httpd                                                              
 9039 nobody    25   0  205m 8344 2176 R  2.0  0.1   0:04.45 httpd                                                              
10529 root      15   0 14076 2436  820 R  2.0  0.0   0:00.23 top                                                                
 7995 nobody    21   0  213m  16m 7220 R  1.6  0.1   0:06.03 httpd                                                              
 8044 nobody    20   0  206m  12m 5024 R  1.6  0.1   0:03.66 httpd                                                              
 8491 nobody    25   0  210m  16m 6772 R  1.6  0.1   0:05.45 httpd                                                              
 8492 nobody    21   0  210m  15m 6240 R  1.6  0.1   0:04.56 httpd                                                              
 8737 nobody    25   0  205m 9640 3380 R  1.6  0.1   0:04.78 httpd                                                              
 8767 nobody    25   0  205m 8344 2176 R  1.6  0.1   0:05.02 httpd                                                              
 8776 nobody    25   0  205m 8452 2264 R  1.6  0.1   0:04.98 httpd                                                              
 8779 nobody    25   0  205m 8520 2340 R  1.6  0.1   0:05.05 httpd                                                              
 8815 nobody    25   0  205m 8452 2272 R  1.6  0.1   0:04.89 httpd                                                              
 8818 nobody    25   0  205m 8456 2276 R  1.6  0.1   0:04.66 httpd                                                              
 8836 nobody    25   0  205m 8444 2272 R  1.6  0.1   0:04.65 httpd                                                              
 8846 nobody    25   0  205m 8428 2240 R  1.6  0.1   0:04.49 httpd                                                              
 8854 nobody    25   0  205m 8376 2200 R  1.6  0.1   0:03.61 httpd                                                              
 8886 nobody    25   0  205m 8344 2176 R  1.6  0.1   0:03.09 httpd   

I already restarted it, and now it's working fine, but this problem is hapenning over and over....

My disks are in RAID. According to these logs, is it really a disk problem?

share|improve this question
    
What Linux version and distribution are you running? –  Mircea Vutcovici Sep 21 '11 at 19:12
    
On what hardware is running the server? What type of disk controller? –  Mircea Vutcovici Sep 21 '11 at 19:14
    
Pleas add the output of the last dmesg messages. –  Mircea Vutcovici Sep 21 '11 at 19:14
    
dmesg doesn't show any disk error. –  valter Sep 21 '11 at 19:34
    
please provide the output of 'df -h'. –  anastrophe Sep 21 '11 at 23:34

2 Answers 2

Check your RAID controller status to see the health of the disks. Use strace on the most I/O intensive threads and see what they are doing. List also all the file descriptors for that process with ls -l /proc/<PID>/fd and try to find out which file is writing.

Use dstat with topio option to find the process that is most I/O expansive. Or you can use iotop

You can investigate further with systemtap to see if it is a FS, disk/controller or user space problem.

share|improve this answer
    
dtrace and topio "command not found" –  valter Sep 21 '11 at 19:33
    
Sorry, I mean dstat. You have to install it. –  Mircea Vutcovici Sep 21 '11 at 19:40

What you have given here are no logs. You only provide server status.

So from you "logs" it looks clean, but this is neither a proof nor a counterproof. Check with disk checking tools, look at the SMART information, watch out for I/O errors or whatever. But don't think that a top output shows disk errors!

Important: Look out for professional help. You asked 18 questions and 8 of them are about the same problem. You provide no information, accept no answer and only rephrase the same question with other words and always blame another culprit.

share|improve this answer
    
I think that the system is using a RAIC controller. So you can not use SMART in this case. You need to use the tools provided by the controller manufacturer. –  Mircea Vutcovici Sep 22 '11 at 13:52
    
@MirceaVutcovici However. This is not the problem. The only problem the OP has is to analyze the problem and wants us to analyze it remotely without giving appropriate information. See the other questions. –  mailq Sep 22 '11 at 14:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.