Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

With random time interval (30 minutes to 7 days) my VirtualBox Centos webserver goes to 100% CPU usage and stops responding to any input (web, ssh, ping, nothing).

All I can do is hard-restart it via VirtualBox and it runs fine for a while until next crash.

Watching TOP output while the crash happens, I can see apache processes jumping to the top of the process list, eating all CPU. Load average spikes to triple digits (from about 0.5 - 1.5 which is normal load).

First I was suspecting some sort of DOS attack, but there is nothing unusual in HTTP request behavior in Apache access_log and no meaningful errors in error_log. Also, after restarting, the server always runs normally, with no excess load.

I made a CRON job that every minute saves TOP output to text file. Unfortunately, it hasn't so far captured the crashed state, presumably because the server is so overloaded that it can't execute CRON.

Here is the last output (~1 minute before the latest crash). I will try to catch a TOP snapshot from a crashed state, but it might take several days and I'll have to keep an ssh session open all the time.

top - 21:01:02 up 7 days, 23:31,  1 user,  load average: 0.03, 0.37, 0.59
Tasks: 221 total,   6 running, 215 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.6%us,  4.7%sy,  0.0%ni, 85.2%id,  0.0%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:   1020132k total,   878044k used,   142088k free,    44552k buffers
Swap:  1146872k total,    62196k used,  1084676k free,   636280k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
21963 apache    20   0  295m 9976 3600 R 59.0  1.0   0:00.57 httpd              
21900 apache    20   0  294m 9392 3644 R 57.1  0.9   0:03.71 httpd              
21946 apache    20   0  294m 8740 3576 R 51.4  0.9   0:00.88 httpd              
21950 apache    20   0  294m 7548 2764 R 49.5  0.7   0:00.56 httpd              
21934 apache    20   0  293m 8032 3620 S 41.9  0.8   0:01.57 httpd              
21958 apache    20   0  295m 9596 3548 R 15.2  0.9   0:00.44 httpd              
21971 root      20   0  9192 1288 1108 S  9.5  0.1   0:00.19 run-parts          
21972 root      20   0 15156 1248  860 R  7.6  0.1   0:00.11 top                
    1 root      20   0 19228  860  728 S  0.0  0.1   0:05.34 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.05 kthreadd           
    3 root      RT   0     0    0    0 S  0.0  0.0   0:15.49 migration/0        
    4 root      20   0     0    0    0 S  0.0  0.0  17:37.36 ksoftirqd/0        
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    6 root      RT   0     0    0    0 S  0.0  0.0   3:16.30 watchdog/0         
    7 root      RT   0     0    0    0 S  0.0  0.0   0:11.17 migration/1        
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1        
    9 root      20   0     0    0    0 S  0.0  0.0  12:34.76 ksoftirqd/1        
   10 root      RT   0     0    0    0 S  0.0  0.0   3:41.83 watchdog/1         
   11 root      RT   0     0    0    0 S  0.0  0.0   0:07.77 migration/2        
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2        
   13 root      20   0     0    0    0 S  0.0  0.0   5:10.25 ksoftirqd/2        
   14 root      RT   0     0    0    0 S  0.0  0.0   3:43.92 watchdog/2         
   15 root      RT   0     0    0    0 S  0.0  0.0   0:05.65 migration/3        
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3        
   17 root      20   0     0    0    0 S  0.0  0.0   3:52.53 ksoftirqd/3        
   18 root      RT   0     0    0    0 S  0.0  0.0   4:12.21 watchdog/3         
   19 root      20   0     0    0    0 S  0.0  0.0   7:22.87 events/0           
   20 root      20   0     0    0    0 S  0.0  0.0   6:38.43 events/1           
   21 root      20   0     0    0    0 S  0.0  0.0   8:45.35 events/2           
   22 root      20   0     0    0    0 S  0.0  0.0  12:50.22 events/3           
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup             
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.01 khelper            
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns              
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.01 async/mgr          
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                 
   28 root      20   0     0    0    0 S  0.0  0.0   0:59.19 sync_supers        
   29 root      20   0     0    0    0 S  0.0  0.0   2:15.75 bdi-default        
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0      
   31 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/1      
   32 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/2      
   33 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/3      
   34 root      20   0     0    0    0 S  0.0  0.0   3:13.88 kblockd/0          
   35 root      20   0     0    0    0 S  0.0  0.0   0:07.92 kblockd/1          
   36 root      20   0     0    0    0 S  0.0  0.0   0:08.61 kblockd/2          
   37 root      20   0     0    0    0 S  0.0  0.0   0:08.81 kblockd/3          
   38 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpid             
   39 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_notify       
   40 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_hotplug      
   41 root      20   0     0    0    0 S  0.0  0.0   0:00.30 ata/0              
   42 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata/1              
   43 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata/2              
   44 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata/3              
   45 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata_aux            
   46 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksuspend_usbd      
   47 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khubd              
   48 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kseriod            
   49 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md/0               
   50 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md/1               
   51 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md/2               
   52 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md/3               
   53 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md_misc/0          
   54 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md_misc/1          
   55 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md_misc/2          
   56 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md_misc/3          
   57 root      20   0     0    0    0 S  0.0  0.0   0:11.19 khungtaskd         
   58 root      20   0     0    0    0 S  0.0  0.0   0:56.26 kswapd0            
   59 root      25   5     0    0    0 S  0.0  0.0   0:00.00 ksmd               
   60 root      39  19     0    0    0 S  0.0  0.0   9:44.81 khugepaged         
   61 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/0              
   62 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/1              
   63 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/2              
   64 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/3              
   65 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/0           
   66 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/1           
   67 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/2           
   68 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/3           
   73 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthrotld/0         
   74 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthrotld/1         
   75 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthrotld/2         
   76 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthrotld/3         
   79 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kpsmoused          
   80 root      20   0     0    0    0 S  0.0  0.0   0:00.00 usbhid_resumer     
  111 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kstriped           
  194 root      20   0     0    0    0 S  0.0  0.0   0:00.32 scsi_eh_0          
  199 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_1          
  234 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_2          
  235 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_3          
  236 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_4          
  237 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_5          
  238 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_6          
  239 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_7          
  240 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_8          
  241 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_9          
  242 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_10         
  243 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_11         
  244 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_12         
  245 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_13         
  246 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_14         
  247 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_15         
  248 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_16         
  249 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_17         
  250 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_18         
  251 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_19         
  252 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_20         
  253 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_21         
  254 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_22         
  255 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_23         
  256 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_24         
  257 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_25         
  258 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_26         
  259 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_27         
  260 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_28         
  261 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_29         
  262 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_30         
  263 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_31         
  359 root      20   0     0    0    0 S  0.0  0.0   1:34.69 kdmflush           
  361 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kdmflush           
  379 root      20   0     0    0    0 S  0.0  0.0   8:30.70 jbd2/dm-0-8        
  380 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  381 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  382 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  383 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  442 root      20   0     0    0    0 S  0.0  0.0   5:44.25 flush-253:0        
  466 root      16  -4 11320  232  232 S  0.0  0.0   0:01.30 udevd              
  790 root      20   0     0    0    0 S  0.0  0.0   0:00.00 jbd2/sda1-8        
  791 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  792 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  793 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  794 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  849 root      20   0     0    0    0 S  0.0  0.0   0:30.81 kauditd            
 1061 root      20   0  9116  412  268 S  0.0  0.0   0:00.38 dhclient           
 1110 root      16  -4 93172  680  548 S  0.0  0.1   2:37.14 auditd             
 1126 root      20   0  243m 4412  792 S  0.0  0.4   0:57.48 rsyslogd           
 1175 root      20   0 64116  420  352 S  0.0  0.0   0:02.16 sshd               
 1211 root      20   0  105m 1268 1268 S  0.0  0.1   0:00.23 mysqld_safe        
 1321 mysql     20   0 1074m  38m 4568 S  0.0  3.9   1873:12 mysqld             
 1413 root      20   0 78720 2408 2320 S  0.0  0.2   1:33.61 master             
 1418 postfix   20   0 78972 2356 2284 S  0.0  0.2   0:14.64 qmgr               
 1423 root      20   0  292m 7896 4588 S  0.0  0.8  49:08.89 httpd              
 1424 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.38 sshd               
 1428 realart   20   0 97860  500  500 S  0.0  0.0   0:00.14 sshd               
 1429 realart   20   0 55352 1472 1472 S  0.0  0.1   0:00.19 sftp-server        
 1439 root      20   0  114m  812  720 S  0.0  0.1   2:44.72 crond              
 1453 root      20   0  4060  472  472 S  0.0  0.0   0:00.05 mingetty           
 1455 root      20   0  4060  472  472 S  0.0  0.0   0:00.03 mingetty           
 1457 root      20   0  4060  472  472 S  0.0  0.0   0:00.03 mingetty           
 1459 root      20   0  4060  472  472 S  0.0  0.0   0:00.06 mingetty           
 1460 root      18  -2 12372  336  336 S  0.0  0.0   0:00.07 udevd              
 1461 root      18  -2 12372  336  336 S  0.0  0.0   0:00.09 udevd              
 1463 root      20   0  4060  472  472 S  0.0  0.0   0:00.04 mingetty           
 1465 root      20   0  4060  472  472 S  0.0  0.0   0:00.02 mingetty           
 1477 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.50 sshd               
 1481 root      20   0 55212 1472 1472 S  0.0  0.1   0:00.24 sftp-server        
 3390 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.37 sshd               
 3394 root      20   0 55212 1472 1472 S  0.0  0.1   0:00.11 sftp-server        
 4653 root      20   0 97864 1996 1996 S  0.0  0.2   0:00.25 sshd               
 4661 realart   20   0 97864  456  456 S  0.0  0.0   0:00.08 sshd               
 4662 realart   20   0  105m 1348 1348 S  0.0  0.1   0:00.14 bash               
 5348 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.28 sshd               
 5352 realart   20   0 97860  500  500 S  0.0  0.0   0:00.08 sshd               
 5353 realart   20   0 55352 1472 1472 S  0.0  0.1   0:00.16 sftp-server        
 5797 root      20   0 97860 1968 1968 S  0.0  0.2   0:00.99 sshd               
 5807 realart   20   0 97860  508  500 S  0.0  0.0   0:00.16 sshd               
 5808 realart   20   0 55344 1472 1472 S  0.0  0.1   0:00.31 sftp-server        
 5996 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.52 sshd               
 6000 realart   20   0 97860  500  500 S  0.0  0.0   0:00.18 sshd               
 6001 realart   20   0 55332 1472 1472 S  0.0  0.1   0:00.26 sftp-server        
 6692 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.43 sshd               
 6697 realart   20   0 97860  500  500 S  0.0  0.0   0:00.08 sshd               
 6698 realart   20   0 55352 1472 1472 S  0.0  0.1   0:00.11 sftp-server        
 7061 root      20   0 97860 1972 1968 S  0.0  0.2   0:00.44 sshd               
 7075 realart   20   0 97860  500  500 S  0.0  0.0   0:00.11 sshd               
 7076 realart   20   0 55388 1400 1396 S  0.0  0.1   0:00.11 sftp-server        
 7805 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.51 sshd               
 7810 realart   20   0 97860  500  500 S  0.0  0.0   0:00.12 sshd               
 7811 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.17 sftp-server        
15364 root      20   0 97860 1972 1968 S  0.0  0.2   0:00.44 sshd               
15370 realart   20   0 97860  508  500 S  0.0  0.0   0:00.12 sshd               
15373 realart   20   0 55344 1480 1472 S  0.0  0.1   0:00.36 sftp-server        
16493 postfix   20   0 78800 3212 2384 S  0.0  0.3   0:00.29 pickup             
17574 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.35 sshd               
17579 realart   20   0 97860  500  500 S  0.0  0.0   0:00.12 sshd               
17580 realart   20   0 55352 1472 1472 S  0.0  0.1   0:00.14 sftp-server        
17731 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.48 sshd               
17739 realart   20   0 97860  500  500 S  0.0  0.0   0:00.15 sshd               
17740 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.17 sftp-server        
19194 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.47 sshd               
19199 realart   20   0 97860  500  500 S  0.0  0.0   0:00.16 sshd               
19201 realart   20   0 55344 1476 1472 S  0.0  0.1   0:00.21 sftp-server        
20124 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.39 sshd               
20128 realart   20   0 97860  500  500 S  0.0  0.0   0:00.16 sshd               
20129 realart   20   0 55352 1472 1472 S  0.0  0.1   0:00.22 sftp-server        
21724 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.27 sshd               
21728 realart   20   0 97860  500  500 S  0.0  0.0   0:00.15 sshd               
21730 realart   20   0 55344 1472 1472 S  0.0  0.1   0:00.35 sftp-server        
21966 root      20   0  136m 1324  976 S  0.0  0.1   0:00.12 crond              
21967 root      20   0  136m 1324  976 S  0.0  0.1   0:00.15 crond              
21970 root      20   0  9192 1172 1012 S  0.0  0.1   0:00.13 bash               
24571 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.38 sshd               
24575 realart   20   0 97860  500  500 S  0.0  0.0   0:00.09 sshd               
24576 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.15 sftp-server        
24887 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.28 sshd               
24891 realart   20   0 97860  500  500 S  0.0  0.0   0:01.11 sshd               
24892 realart   20   0 55348 1476 1476 S  0.0  0.1   0:05.39 sftp-server        
25529 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.54 sshd               
25535 realart   20   0 97860  500  500 S  0.0  0.0   0:00.19 sshd               
25536 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.27 sftp-server        
27093 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.47 sshd               
27098 realart   20   0 97860  500  500 S  0.0  0.0   0:00.14 sshd               
27099 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.15 sftp-server        
27395 root      20   0 97860 1960 1960 S  0.0  0.2   0:00.45 sshd               
27399 realart   20   0 97860  500  500 S  0.0  0.0   0:00.16 sshd               
27400 realart   20   0 55332 1472 1472 S  0.0  0.1   0:00.21 sftp-server        
29292 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.39 sshd               
29300 realart   20   0 97860  500  500 S  0.0  0.0   0:00.06 sshd               
29301 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.11 sftp-server        
29715 root      20   0 97860 1964 1964 S  0.0  0.2   0:00.47 sshd               
29720 realart   20   0 97860  500  500 S  0.0  0.0   0:00.08 sshd               
29722 realart   20   0 55212 1472 1472 S  0.0  0.1   0:00.16 sftp-server        
32086 root      20   0 97860 1988 1968 S  0.0  0.2   0:00.61 sshd               
32092 root      20   0 55212 1480 1476 S  0.0  0.1   0:00.31 sftp-server
share|improve this question
    
provided information is not enough. only thing i can say right now is that you don't have any problems with disk performance, this is redhat-like system (RHEL6/Centos6/Fedora 16+), 4vCPU's, you are running apache prefork. Please clarify your apache configuration. rpm -V httpd, rpm -q httpd. check for netstat -an and dmesg with cron, update all packages. –  GioMac Aug 14 '13 at 23:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.